[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 508449247 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002322] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.007797] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010125] LSM: Security Framework initializing [ 0.011045] Yama: becoming mindful. [ 0.012034] SELinux: Initializing. [ 0.013059] *** VALIDATE selinux *** [ 0.021276] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026068] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027127] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029010] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030109] *** VALIDATE tmpfs *** [ 0.033107] *** VALIDATE proc *** [ 0.034233] *** VALIDATE cgroup *** [ 0.035010] *** VALIDATE cgroup2 *** [ 0.036254] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037144] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039029] Spectre V2 : User space: Vulnerable [ 0.040011] Speculative Store Bypass: Vulnerable [ 0.043588] debug: unmapping init [mem 0xffffffffa1059000-0xffffffffa1060fff] [ 0.045157] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046728] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047022] ... version: 2 [ 0.048012] ... bit width: 48 [ 0.049013] ... generic registers: 4 [ 0.050015] ... value mask: 0000ffffffffffff [ 0.051014] ... max period: 00007fffffffffff [ 0.052012] ... fixed-purpose events: 3 [ 0.053013] ... event mask: 000000070000000f [ 0.054262] rcu: Hierarchical SRCU implementation. [ 0.056457] smp: Bringing up secondary CPUs ... [ 0.057605] x86: Booting SMP configuration: [ 0.058025] .... node #0, CPUs: #1 #2 #3 [ 0.063689] smp: Brought up 1 node, 4 CPUs [ 0.065025] smpboot: Max logical packages: 1 [ 0.066015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.149783] node 0 deferred pages initialised in 82ms [ 0.153095] devtmpfs: initialized [ 0.154277] x86/mm: Memory block size: 128MB [ 0.158789] gcov: version magic: 0x41383552 [ 0.161188] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.165075] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.168303] pinctrl core: initialized pinctrl subsystem [ 0.170146] [ 0.170818] ************************************************************* [ 0.173014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.176013] ** ** [ 0.178011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.181013] ** ** [ 0.184014] ** This means that this kernel is built to expose internal ** [ 0.186013] ** IOMMU data structures, which may compromise security on ** [ 0.189012] ** your system. ** [ 0.191009] ** ** [ 0.193014] ** If you see this message and you are not debugging the ** [ 0.196012] ** kernel, report this immediately to your vendor! ** [ 0.198011] ** ** [ 0.201015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.203011] ************************************************************* [ 0.205666] NET: Registered protocol family 16 [ 0.208487] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.211059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.214067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.218012] cpuidle: using governor menu [ 0.220147] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.222427] PCI: Using configuration type 1 for base access [ 0.225135] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.233105] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.236061] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.240180] cryptd: max_cpu_qlen set to 1000 [ 0.242528] ACPI: Added _OSI(Module Device) [ 0.243000] ACPI: Added _OSI(Processor Device) [ 0.243000] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.243000] ACPI: Added _OSI(Processor Aggregator Device) [ 0.248680] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.255514] ACPI: Interpreter enabled [ 0.258071] ACPI: PM: (supports S0 S3 S4 S5) [ 0.259013] ACPI: Using IOAPIC for interrupt routing [ 0.262116] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.265372] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.274627] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.277036] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.280016] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.283087] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.289375] acpiphp: Slot [2] registered [ 0.291124] acpiphp: Slot [5] registered [ 0.292115] acpiphp: Slot [6] registered [ 0.294104] acpiphp: Slot [7] registered [ 0.296130] acpiphp: Slot [8] registered [ 0.298141] acpiphp: Slot [9] registered [ 0.299089] acpiphp: Slot [10] registered [ 0.300095] acpiphp: Slot [3] registered [ 0.302105] acpiphp: Slot [4] registered [ 0.303115] acpiphp: Slot [11] registered [ 0.305106] acpiphp: Slot [12] registered [ 0.307094] acpiphp: Slot [13] registered [ 0.308099] acpiphp: Slot [14] registered [ 0.309102] acpiphp: Slot [15] registered [ 0.311063] acpiphp: Slot [16] registered [ 0.312083] acpiphp: Slot [17] registered [ 0.313109] acpiphp: Slot [18] registered [ 0.315108] acpiphp: Slot [19] registered [ 0.317111] acpiphp: Slot [20] registered [ 0.319131] acpiphp: Slot [21] registered [ 0.321108] acpiphp: Slot [22] registered [ 0.322216] acpiphp: Slot [23] registered [ 0.324122] acpiphp: Slot [24] registered [ 0.326123] acpiphp: Slot [25] registered [ 0.328114] acpiphp: Slot [26] registered [ 0.330107] acpiphp: Slot [27] registered [ 0.332130] acpiphp: Slot [28] registered [ 0.333158] acpiphp: Slot [29] registered [ 0.335198] acpiphp: Slot [30] registered [ 0.337136] acpiphp: Slot [31] registered [ 0.339063] PCI host bridge to bus 0000:00 [ 0.340016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.342017] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.344019] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.347024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.349019] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.352025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.354163] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.357933] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.360914] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.370014] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.374675] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.377017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.379016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.381018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.384508] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.387693] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.390033] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.392687] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.397010] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.411978] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.417014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.424381] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.432015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.440015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.458026] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.472822] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.478017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.491016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.513013] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.529486] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.540016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.552014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.568021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.578396] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.586014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.593014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.609019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.619541] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.626016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.636015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.652075] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.661854] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.672023] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.684023] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.703021] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.715528] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.717372] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.719328] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.721344] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.724225] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.728130] iommu: Default domain type: Passthrough [ 0.730488] SCSI subsystem initialized [ 0.732160] ACPI: bus type USB registered [ 0.733181] usbcore: registered new interface driver usbfs [ 0.736071] usbcore: registered new interface driver hub [ 0.738079] usbcore: registered new device driver usb [ 0.739164] pps_core: LinuxPPS API ver. 1 registered [ 0.741023] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.744055] PTP clock support registered [ 0.746079] EDAC MC: Ver: 3.0.0 [ 0.748150] PCI: Using ACPI for IRQ routing [ 0.749716] NetLabel: Initializing [ 0.752014] NetLabel: domain hash size = 128 [ 0.754012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.756078] NetLabel: unlabeled traffic allowed by default [ 0.759004] vgaarb: loaded [ 0.760342] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.762026] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.772061] clocksource: Switched to clocksource kvm-clock [ 0.887319] VFS: Disk quotas dquot_6.6.0 [ 0.888931] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.891378] *** VALIDATE ramfs *** [ 0.892663] *** VALIDATE hugetlbfs *** [ 0.894085] pnp: PnP ACPI init [ 0.896295] pnp: PnP ACPI: found 6 devices [ 0.916149] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.919292] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.921480] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.923850] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.926429] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.928693] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.931366] NET: Registered protocol family 2 [ 0.933783] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.938802] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.942397] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.947300] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.950540] TCP: Hash tables configured (established 65536 bind 65536) [ 0.953305] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.956045] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.958733] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.963158] NET: Registered protocol family 1 [ 0.966181] RPC: Registered named UNIX socket transport module. [ 0.968482] RPC: Registered udp transport module. [ 0.970330] RPC: Registered tcp transport module. [ 0.972247] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.974778] NET: Registered protocol family 44 [ 0.976657] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.978944] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.981292] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.983787] PCI: CLS 0 bytes, default 64 [ 0.986314] Unpacking initramfs... [ 2.370777] debug: unmapping init [mem 0xffff96f53cc54000-0xffff96f53ffbffff] [ 2.374748] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.376807] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.380607] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.884437] Initialise system trusted keyrings [ 2.885990] Key type blacklist registered [ 2.887645] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.897687] zbud: loaded [ 2.900906] *** VALIDATE nfs *** [ 2.902422] *** VALIDATE nfs4 *** [ 2.904136] pstore: using deflate compression [ 2.908173] Platform Keyring initialized [ 3.012502] NET: Registered protocol family 38 [ 3.013720] Key type asymmetric registered [ 3.014906] Asymmetric key parser 'x509' registered [ 3.016343] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.020200] io scheduler mq-deadline registered [ 3.022459] io scheduler kyber registered [ 3.024276] io scheduler bfq registered [ 3.026525] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.029019] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.032156] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.034190] ACPI: Power Button [PWRF] [ 3.038791] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.044731] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.060037] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.067272] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.085691] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.114373] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.144381] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.150987] Non-volatile memory driver v1.3 [ 3.152835] Linux agpgart interface v0.103 [ 3.180080] virtio_blk virtio1: [vda] 150008 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.182483] vda: detected capacity change from 0 to 76804096 [ 3.198220] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.201733] vdb: detected capacity change from 0 to 1073741824 [ 3.216034] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.219056] vdc: detected capacity change from 0 to 2621440000 [ 3.234656] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.237729] vdd: detected capacity change from 0 to 2621440000 [ 3.251177] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.253630] vde: detected capacity change from 0 to 4294967296 [ 3.271891] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.274404] vdf: detected capacity change from 0 to 4294967296 [ 3.279479] libphy: Fixed MDIO Bus: probed [ 3.297122] usbcore: registered new interface driver usbserial_generic [ 3.300041] usbserial: USB Serial support registered for generic [ 3.302901] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.307684] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.309095] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.311537] mousedev: PS/2 mouse device common for all mice [ 3.314529] rtc_cmos 00:05: RTC can wake from S4 [ 3.317654] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.320297] rtc_cmos 00:05: registered as rtc0 [ 3.326712] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.329717] intel_pstate: CPU model not supported [ 3.334091] hid: raw HID events driver (C) Jiri Kosina [ 3.336905] usbcore: registered new interface driver usbhid [ 3.337209] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.339392] usbhid: USB HID core driver [ 3.346921] drop_monitor: Initializing network drop monitor service [ 3.350260] Initializing XFRM netlink socket [ 3.352470] NET: Registered protocol family 10 [ 3.355725] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.358192] Segment Routing with IPv6 [ 3.360838] NET: Registered protocol family 17 [ 3.363115] mpls_gso: MPLS GSO support [ 3.370250] RAS: Correctable Errors collector initialized. [ 3.372565] AVX version of gcm_enc/dec engaged. [ 3.374033] AES CTR mode by8 optimization enabled [ 3.454575] sched_clock: Marking stable (3454553841, 0)->(4332962559, -878408718) [ 3.458079] registered taskstats version 1 [ 3.460161] Loading compiled-in X.509 certificates [ 3.461906] zswap: loaded using pool lzo/zbud [ 3.483995] Key type big_key registered [ 3.496383] Key type encrypted registered [ 3.498346] ima: No TPM chip found, activating TPM-bypass! [ 3.501600] ima: Allocated hash algorithm: sha1 [ 3.504397] ima: No architecture policies found [ 3.507023] evm: Initialising EVM extended attributes: [ 3.510177] evm: security.selinux [ 3.511847] evm: security.ima [ 3.513656] evm: security.capability [ 3.515807] evm: HMAC attrs: 0x1 [ 3.519237] rtc_cmos 00:05: setting system clock to 2026-09-15 04:45:28 UTC (1789447528) [ 3.527593] debug: unmapping init [mem 0xffffffffa2003000-0xffffffffa21fffff] [ 3.531647] debug: unmapping init [mem 0xffffffffa0d82000-0xffffffffa1058fff] [ 3.541082] Write protecting the kernel read-only data: 28672k [ 3.545774] debug: unmapping init [mem 0xffffffff9f403000-0xffffffff9f5fffff] [ 3.550240] debug: unmapping init [mem 0xffffffff9fd14000-0xffffffff9fdfffff] [ 3.584842] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.596304] systemd[1]: Detected virtualization kvm. [ 3.598846] systemd[1]: Detected architecture x86-64. [ 3.601658] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.631716] systemd[1]: No hostname configured. [ 3.633824] systemd[1]: Set hostname to . [ 3.636070] random: systemd: uninitialized urandom read (16 bytes read) [ 3.639040] systemd[1]: Initializing machine ID from random generator. [ 3.767354] random: systemd: uninitialized urandom read (16 bytes read) [ 3.770879] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.777694] random: systemd: uninitialized urandom read (16 bytes read) [ 3.781099] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.785813] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Journal Service... Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.346100] device-mapper: uevent: version 1.0.3 [ 4.348532] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.027733] virtio_net virtio0 ens2: renamed from eth0 [ 5.067950] random: fast init done [ 5.077657] scsi host0: ata_piix [ 5.112860] scsi host1: ata_piix [ 5.114496] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.116936] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.118331] dracut-initqueue[591]: RTNETLINK answers: File exists [ 10.022173] random: crng init done [ 10.023685] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.230935] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.396307] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.633691] SELinux: Disabled at runtime. [ 11.693460] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.702896] systemd[1]: Detected virtualization kvm. [ 11.704955] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.168241] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.171890] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.176674] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.181246] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.184789] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.191425] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.204901] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... [ 12.237767] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. Mounting Huge Pages File System... [ OK ] Created slice system-getty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Apply Kernel Variables... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Paths. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 12.670943] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 12.995247] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.002147] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.096450] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.127780] EDAC sbridge: Ver: 1.1.2 [ 14.510287] Key type dns_resolver registered [ 14.820339] NFS: Registering the id_resolver key type [ 14.822051] Key type id_resolver registered [ 14.823722] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg327-server login: [ 42.107006] hrtimer: interrupt took 2041725 ns [ 47.854826] libcfs: loading out-of-tree module taints kernel. [ 47.947920] Key type ._llcrypt registered [ 47.951554] Key type .llcrypt registered [ 48.090430] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [ 67.260079] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 68.850421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 68.891166] alg: No test for adler32 (adler32-zlib) [ 70.332530] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 71.292700] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 73.071223] Key type lgssc registered [ 75.361733] Lustre: Echo OBD driver; http://www.lustre.org/ [ 90.284866] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 129.896539] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 142.830378] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 142.868956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 144.148142] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 144.182652] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 144.303965] Lustre: lustre-MDT0000: new disk, initializing [ 144.403490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 144.424390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 148.354961] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 161.068559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 161.221692] Lustre: 6513:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 161.263575] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 161.267606] Lustre: Skipped 1 previous similar message [ 161.365723] Lustre: lustre-MDT0001: new disk, initializing [ 161.448454] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 161.487887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 161.505854] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 165.731992] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 170.049570] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 178.761207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 179.020251] Lustre: lustre-OST0000: new disk, initializing [ 179.025670] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 179.033905] Lustre: 8451:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 179.130139] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 179.665109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 179.682295] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 179.740459] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 185.700582] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 200.657325] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 200.849405] Lustre: lustre-OST0001: new disk, initializing [ 200.853835] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 200.860879] Lustre: 9526:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 200.965899] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 208.337587] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 208.451111] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 208.463765] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 208.510690] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 220.617650] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 228.902836] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 236.646845] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing check_logdir /tmp/testlogs/ [ 242.984866] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing yml_node [ 248.906489] Lustre: DEBUG MARKER: Client: 2.17.58.40 [ 252.057783] Lustre: DEBUG MARKER: MDS: 2.17.58.40 [ 255.209841] Lustre: DEBUG MARKER: OSS: 2.17.58.40 [ 257.151789] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 15 00:49:40 EDT 2026 [ 277.334394] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 279.327412] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 299.999976] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 300.015483] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 300.047356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 300.516051] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 300.538107] Lustre: Skipped 2 previous similar messages [ 300.554409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 300.561069] Lustre: Skipped 3 previous similar messages [ 301.931610] Lustre: server umount lustre-MDT0000 complete [ 305.564771] LustreError: 6505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789447830 with bad export cookie 1942082147883193623 [ 305.570809] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 305.574725] LustreError: 6505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 305.631630] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 305.632349] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 305.641208] Lustre: Skipped 1 previous similar message [ 305.651832] Lustre: Skipped 1 previous similar message [ 310.752418] LustreError: 6521:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 310.752526] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 310.790452] LustreError: 6521:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 311.962439] Lustre: server umount lustre-MDT0001 complete [ 315.397839] Lustre: server umount lustre-OST0000 complete [ 318.961534] Lustre: server umount lustre-OST0001 complete [ 325.933185] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [ 334.935514] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 374.922112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 375.187628] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 375.221695] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 375.313734] Lustre: lustre-MDT0000: new disk, initializing [ 375.391188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 375.422853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 381.178948] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 392.614262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 392.822576] Lustre: 15079:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 392.925910] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 392.929652] Lustre: Skipped 1 previous similar message [ 393.070939] Lustre: lustre-MDT0001: new disk, initializing [ 393.210140] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 393.283366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 393.302050] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 396.419650] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 405.580356] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 410.817877] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 416.585721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 416.898378] Lustre: lustre-OST0000: new disk, initializing [ 416.901743] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 416.912856] Lustre: 17045:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 417.003386] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 418.446077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 418.462392] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 418.588656] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 422.884423] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 431.878374] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 436.787685] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 437.008929] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 442.203191] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 442.454260] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 449.507976] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 449.508589] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 449.526976] LustreError: Skipped 1 previous similar message [ 449.528435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 449.528442] Lustre: Skipped 1 previous similar message [ 449.571232] Lustre: Skipped 1 previous similar message [ 450.943033] Lustre: server umount lustre-OST0000 complete [ 454.624126] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 454.626245] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 454.643243] Lustre: Skipped 1 previous similar message [ 459.746370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 459.752389] Lustre: Skipped 4 previous similar messages [ 460.822826] Lustre: server umount lustre-MDT0000 complete [ 464.565767] LustreError: 15072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789447989 with bad export cookie 1942082147883195156 [ 464.573279] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 465.090254] Lustre: server umount lustre-MDT0001 complete [ 469.118314] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 00:53:12 (1789447992) [ 472.047664] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 474.326865] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 00:53:17 (1789447997) [ 481.029418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 481.667800] LustreError: 19456:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 481.737452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 485.960774] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 486.890958] LustreError: 19457:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 492.009614] LustreError: 19456:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 493.967228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 498.639944] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 505.439959] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 510.662832] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 517.094142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 523.500986] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 529.984512] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 537.056986] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 537.070895] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 537.083745] Lustre: Skipped 1 previous similar message [ 543.199876] Lustre: server umount lustre-OST0000 complete [ 553.623413] Lustre: server umount lustre-MDT0000 complete [ 557.539844] LustreError: 19457:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 557.564426] LustreError: 19457:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 557.821086] LustreError: 22237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789448082 with bad export cookie 1942082147883196038 [ 557.841135] LustreError: 22237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 557.849883] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 558.229942] Lustre: server umount lustre-MDT0001 complete [ 569.420453] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 572.144856] Key type lgssc unregistered [ 572.439613] LNet: 22989:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 572.453988] LNetError: 22989:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 572.478719] LNet: Removed LNI 192.168.203.127@tcp [ 573.412487] Key type .llcrypt unregistered [ 573.415681] Key type ._llcrypt unregistered [ 595.229885] Key type ._llcrypt registered [ 595.233667] Key type .llcrypt registered [ 595.335540] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 596.774721] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 596.799833] alg: No test for adler32 (adler32-zlib) [ 597.896573] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 598.143529] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 599.802195] Key type lgssc registered [ 600.953310] Lustre: Echo OBD driver; http://www.lustre.org/ [ 608.455837] loop: module loaded [ 608.540957] loop0: detected capacity change from 0 to 204800000 [ 608.732059] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 608.753221] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 609.146665] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 614.501196] loop0: detected capacity change from 0 to 204800000 [ 614.756991] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 614.778120] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 614.804966] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 619.822508] loop0: detected capacity change from 0 to 204800000 [ 625.678991] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 625.706735] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 627.004550] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 627.093973] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 627.254778] Lustre: lustre-MDT0000: new disk, initializing [ 627.357132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 627.382648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 632.187134] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 641.488099] loop1: detected capacity change from 0 to 204800000 [ 647.857752] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 648.231624] Lustre: lustre-OST07c6: new disk, initializing [ 648.251734] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 648.258942] Lustre: Skipped 1 previous similar message [ 648.269750] Lustre: 26170:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 648.344303] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 654.561986] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 657.002727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 657.014597] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 657.058672] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 659.092558] Lustre: 25156:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 659.099629] Lustre: 25156:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 659.113159] Lustre: 25156:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 659.121902] Lustre: 25156:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 659.129743] Lustre: 25156:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 659.136586] Lustre: 25156:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 674.985967] Lustre: 25157:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 674.996776] Lustre: 25157:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 675.013979] Lustre: 25157:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 675.022939] Lustre: 25157:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 675.030848] Lustre: 25157:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 675.040710] Lustre: 25157:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 680.252035] Lustre: 25155:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 680.266720] Lustre: 25155:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 680.273413] Lustre: 25155:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 680.281978] Lustre: 25155:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 680.292196] Lustre: 25155:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 680.303394] Lustre: 25155:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 680.312483] Lustre: 25155:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 680.321294] Lustre: 25155:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 680.325984] Lustre: 25155:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 680.332137] Lustre: 25155:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 680.338527] Lustre: 25155:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 680.345688] Lustre: 25155:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 683.596726] Lustre: 25156:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 683.614550] Lustre: 25156:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 683.629294] Lustre: 25156:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 683.638252] Lustre: 25156:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 683.646545] Lustre: 25156:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 683.659650] Lustre: 25156:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 683.675513] Lustre: 25156:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 683.682724] Lustre: 25156:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 683.689226] Lustre: 25156:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 683.699927] Lustre: 25156:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 683.708796] Lustre: 25156:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 683.721132] Lustre: 25156:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 688.600557] Lustre: 25156:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 688.608867] Lustre: 25156:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 688.618545] Lustre: 25156:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 688.627231] Lustre: 25156:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 688.634970] Lustre: 25156:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 688.642443] Lustre: 25156:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 688.651533] Lustre: 25156:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 688.664871] Lustre: 25156:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 688.673989] Lustre: 25156:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 688.679977] Lustre: 25156:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 688.690407] Lustre: 25156:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 688.696643] Lustre: 25156:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 700.263233] Lustre: 25157:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 700.275534] Lustre: 25157:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 700.281303] Lustre: 25157:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 700.288101] Lustre: 25157:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 700.296490] Lustre: 25157:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 700.305398] Lustre: 25157:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 700.312863] Lustre: 25157:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 700.319706] Lustre: 25157:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 700.324932] Lustre: 25157:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 700.329354] Lustre: 25157:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 700.334921] Lustre: 25157:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 700.339607] Lustre: 25157:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 749.024858] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 749.046986] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 749.076516] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 753.780636] Lustre: server umount lustre-OST07c6 complete [ 763.195634] Lustre: server umount lustre-MDT0000 complete [ 779.162542] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 781.848524] Key type lgssc unregistered [ 782.100736] LNet: 28725:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 782.123473] LNetError: 28725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 782.147388] LNet: Removed LNI 192.168.203.127@tcp [ 783.076851] Key type .llcrypt unregistered [ 783.079057] Key type ._llcrypt unregistered [ 801.888610] Key type ._llcrypt registered [ 801.891677] Key type .llcrypt registered [ 802.004761] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 803.036232] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 803.172160] alg: No test for adler32 (adler32-zlib) [ 804.273965] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 804.501525] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 806.215544] Key type lgssc registered [ 807.267745] Lustre: Echo OBD driver; http://www.lustre.org/ [ 815.450518] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 815.472154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 816.966328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 821.690502] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 826.343751] LustreError: 29994:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 826.362819] LustreError: 29994:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 830.323394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 830.704342] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 834.712568] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 841.716898] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 846.454702] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 852.950301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 853.425539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 859.329962] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 861.621323] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 867.590566] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 886.243396] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 886.251203] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 886.267350] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 887.272158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 887.276815] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 887.283389] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 888.840324] Lustre: server umount lustre-OST0000 complete [ 897.504468] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 897.507744] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 897.533091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 897.541027] Lustre: Skipped 2 previous similar messages [ 899.006255] Lustre: server umount lustre-MDT0000 complete [ 902.228886] LustreError: 31839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789448427 with bad export cookie 16845350064008121717 [ 902.232370] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 902.242577] LustreError: 31839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 902.477897] Lustre: server umount lustre-MDT0001 complete [ 912.724329] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 915.378858] Key type lgssc unregistered [ 915.697938] LNet: 33731:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 915.711630] LNetError: 33731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 916.773630] LNet: Removed LNI 192.168.203.127@tcp [ 917.672393] Key type .llcrypt unregistered [ 917.675285] Key type ._llcrypt unregistered [ 936.929715] Key type ._llcrypt registered [ 936.939592] Key type .llcrypt registered [ 937.111244] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 938.365724] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 938.467894] alg: No test for adler32 (adler32-zlib) [ 939.513931] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 939.768063] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 941.455215] Key type lgssc registered [ 942.795700] Lustre: Echo OBD driver; http://www.lustre.org/ [ 951.038582] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 951.059216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 952.765307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 957.935477] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 963.064024] LustreError: 35011:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 963.110937] LustreError: 35011:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 967.562153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 968.230712] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 973.537956] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 979.383665] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 985.315578] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 991.534712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 991.997066] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 998.166904] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 998.475683] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1006.298535] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1018.852584] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1018.860424] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1018.877574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1021.973730] Lustre: server umount lustre-OST0000 complete [ 1029.092468] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1029.106531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1029.112594] Lustre: Skipped 2 previous similar messages [ 1029.123043] Lustre: Skipped 2 previous similar messages [ 1031.383426] Lustre: server umount lustre-MDT0000 complete [ 1034.225803] LustreError: 35011:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1035.067530] LustreError: 34992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789448560 with bad export cookie 16208162407600165937 [ 1035.090362] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1035.368939] Lustre: server umount lustre-MDT0001 complete [ 1044.788951] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 1047.386928] Key type lgssc unregistered [ 1047.700580] LNet: 38541:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1047.709823] LNetError: 38541:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1047.727042] LNet: Removed LNI 192.168.203.127@tcp [ 1048.792431] Key type .llcrypt unregistered [ 1048.795378] Key type ._llcrypt unregistered [ 1074.211259] Key type ._llcrypt registered [ 1074.216787] Key type .llcrypt registered [ 1074.318391] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 1075.327933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1075.454075] alg: No test for adler32 (adler32-zlib) [ 1076.572763] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 1076.878573] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 1078.511231] Key type lgssc registered [ 1079.748960] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1087.090683] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1087.112322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1088.530690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1092.957760] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1098.736371] LustreError: 39956:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1098.762124] LustreError: 39956:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1102.135813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1102.602120] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1107.048316] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1114.279811] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1119.837916] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1126.361945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1126.661181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1133.317695] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1133.813061] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1141.646606] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1148.208768] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.27@tcp (stopping) [ 1148.818401] Lustre: server umount lustre-MDT0000 complete [ 1148.896922] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1148.909125] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1148.921779] LustreError: 39955:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1149.410474] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1149.424825] Lustre: Skipped 1 previous similar message [ 1151.986712] LustreError: 39935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789448676 with bad export cookie 10110473367348795337 [ 1151.999397] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1151.999633] LustreError: 39935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1153.298608] LustreError: 42448:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1153.314881] LustreError: 42448:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1153.329927] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.27@tcp (stopping) [ 1154.528606] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1154.548619] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1156.584235] LustreError: 39952:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1156.584241] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1156.584251] LustreError: 39952:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1161.701352] LustreError: 39951:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1161.701517] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1161.726803] LustreError: 39951:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1161.746116] Lustre: Skipped 1 previous similar message [ 1166.305812] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1166.610090] Lustre: server umount lustre-MDT0001 complete [ 1187.455345] Lustre: server umount lustre-OST0000 complete [ 1203.782992] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 1206.667961] Key type lgssc unregistered [ 1206.964124] LNet: 43813:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1206.969433] LNetError: 43813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1206.987665] LNet: Removed LNI 192.168.203.127@tcp [ 1207.943805] Key type .llcrypt unregistered [ 1207.947038] Key type ._llcrypt unregistered [ 1234.720490] Key type ._llcrypt registered [ 1234.722597] Key type .llcrypt registered [ 1234.802371] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 1235.779923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1235.902074] alg: No test for adler32 (adler32-zlib) [ 1237.219176] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 1237.554517] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 1239.272898] Key type lgssc registered [ 1240.198937] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1249.349401] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1249.382270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1250.954902] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1255.680842] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1260.512911] LustreError: 45228:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1260.542814] LustreError: 45228:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1264.819252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1265.152497] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1270.529586] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1279.248260] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1285.156164] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1292.166543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1292.548932] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1295.611270] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1299.622774] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1307.703632] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1320.751197] Lustre: lustre-OST0000: Not available for connect from 192.168.203.27@tcp (stopping) [ 1321.445782] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1321.447358] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1321.495802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1321.500635] LustreError: Skipped 1 previous similar message [ 1321.520737] Lustre: Skipped 1 previous similar message [ 1323.978790] Lustre: server umount lustre-OST0000 complete [ 1347.041722] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1347.055969] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1347.066252] Lustre: Skipped 1 previous similar message [ 1347.067776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1347.111966] Lustre: Skipped 1 previous similar message [ 1352.167290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1352.182694] Lustre: Skipped 1 previous similar message [ 1357.279522] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1357.280567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1357.313878] Lustre: Skipped 1 previous similar message [ 1357.626290] Lustre: server umount lustre-MDT0000 complete [ 1361.376761] LustreError: 45222:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1361.393036] LustreError: 45222:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1362.108682] LustreError: 45207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789448887 with bad export cookie 12528531069567144269 [ 1362.112521] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1362.125756] LustreError: 45207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1368.649198] Lustre: server umount lustre-MDT0001 complete [ 1380.479681] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 1383.597567] Key type lgssc unregistered [ 1383.833511] LNet: 48938:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1383.843206] LNetError: 48938:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1383.865241] LNet: Removed LNI 192.168.203.127@tcp [ 1385.054640] Key type .llcrypt unregistered [ 1385.056499] Key type ._llcrypt unregistered [ 1415.677592] Key type ._llcrypt registered [ 1415.680531] Key type .llcrypt registered [ 1415.873511] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 1417.295732] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1417.320493] alg: No test for adler32 (adler32-zlib) [ 1418.330255] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 1418.480359] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 1420.127743] Key type lgssc registered [ 1421.483647] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1430.375370] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1430.408385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1431.985677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1436.544837] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1441.263944] LustreError: 50354:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1441.281349] LustreError: 50354:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1444.799643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1445.203964] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1449.410499] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1456.731372] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1461.711517] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1467.561667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1467.848098] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1473.050529] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1476.088176] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1479.562745] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1484.441568] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1487.515384] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1499.619940] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-15 1h10m22s [ 1501.372761] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1506.602281] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.27@tcp (stopping) [ 1506.786478] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1506.787289] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1507.664777] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.27@tcp (stopping) [ 1507.674580] Lustre: Skipped 3 previous similar messages [ 1509.220871] LustreError: 53252:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (000000005553d04f) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 1509.512932] Lustre: server umount lustre-MDT0000 complete [ 1512.722107] LustreError: 50351:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1513.039287] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1513.962687] LustreError: 50351:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1514.000139] LustreError: 50351:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1518.863526] LustreError: 51593:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1519.823793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1519.957957] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1520.131580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1520.241201] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1521.203189] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1524.203689] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1527.778130] Lustre: DEBUG MARKER: Wait for df (36716) ... [ 1529.510739] Lustre: DEBUG MARKER: done [ 1535.456390] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1535.472767] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1535.492110] Lustre: Skipped 2 previous similar messages [ 1535.502941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1535.508457] Lustre: Skipped 3 previous similar messages [ 1536.481548] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1541.603754] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1541.613397] Lustre: Skipped 2 previous similar messages [ 1546.724670] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1546.730739] Lustre: Skipped 2 previous similar messages [ 1548.255547] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1548.455631] Lustre: server umount lustre-OST0000 complete [ 1556.969153] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1556.973935] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1556.974882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1556.989129] Lustre: Skipped 1 previous similar message [ 1558.185198] Lustre: server umount lustre-MDT0000 complete [ 1561.469270] LustreError: 52196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789449086 with bad export cookie 16230112421422276385 [ 1561.478120] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1561.491586] LustreError: 52196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1561.885904] Lustre: server umount lustre-MDT0001 complete [ 1572.488340] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 1575.419923] Key type lgssc unregistered [ 1575.737494] LNet: 55280:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1575.765915] LNetError: 55280:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1575.788188] LNet: Removed LNI 192.168.203.127@tcp [ 1576.749470] Key type .llcrypt unregistered [ 1576.754898] Key type ._llcrypt unregistered [ 1614.761963] Key type ._llcrypt registered [ 1614.769658] Key type .llcrypt registered [ 1614.946541] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 1616.126665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1616.183579] alg: No test for adler32 (adler32-zlib) [ 1617.399414] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 1617.684508] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 1619.319328] Key type lgssc registered [ 1620.695850] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1630.058863] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1630.082776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1631.348972] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1631.386681] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1631.582617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1636.125512] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1644.204601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1644.348834] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1644.379880] Lustre: 57484:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1644.534717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1648.935807] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1657.441697] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1662.088216] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1668.845380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1669.034785] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1669.249661] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1674.928216] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1675.849485] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1683.501496] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1689.584489] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-15 1h13m32s [ 1691.063765] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1697.399103] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1699.749613] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1699.791204] LustreError: 59220:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1720.122592] Lustre: lustre-MDT0000: Client e8dc2ea1-611c-4419-91b4-b7e275010681 (at 192.168.203.27@tcp) reconnecting [ 1739.823111] LustreError: 59220:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1741.528709] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1743.703576] Lustre: DEBUG MARKER: done [ 1751.520742] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1751.531639] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1751.544315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1752.033756] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1752.043390] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1752.063930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1752.070972] Lustre: Skipped 1 previous similar message [ 1755.150240] Lustre: server umount lustre-OST0000 complete [ 1762.274932] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1762.275546] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1762.284346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1765.474392] Lustre: server umount lustre-MDT0000 complete [ 1768.967760] LustreError: 60244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789449293 with bad export cookie 2212133294027395035 [ 1768.979534] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1768.985208] LustreError: 60244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1769.437468] Lustre: server umount lustre-MDT0001 complete [ 1780.418296] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 1783.344541] Key type lgssc unregistered [ 1783.679097] LNet: 61812:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1783.692675] LNetError: 61812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1783.710289] LNet: Removed LNI 192.168.203.127@tcp [ 1784.497233] Key type .llcrypt unregistered [ 1784.502566] Key type ._llcrypt unregistered [ 1827.215630] Key type ._llcrypt registered [ 1827.219017] Key type .llcrypt registered [ 1827.356086] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 1828.287277] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1828.474499] alg: No test for adler32 (adler32-zlib) [ 1829.525766] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 1829.792420] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 1831.431586] Key type lgssc registered [ 1832.416493] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1835.367374] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1835.390984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1836.689298] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1836.719304] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1836.879153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1837.674434] Lustre: Failing over lustre-MDT0000 [ 1837.969942] Lustre: server umount lustre-MDT0000 complete [ 1843.096861] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 01:16:06 (1789449366) [ 1848.996817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1849.310788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1853.450507] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1861.528803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1861.626752] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1861.681771] Lustre: 64603:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1862.066507] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1867.634066] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1875.959894] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1881.846649] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1888.765212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1889.067353] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1889.308884] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1895.580755] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1897.128291] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1902.301472] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1908.168863] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1917.921721] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1917.928570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1917.940418] Lustre: Skipped 1 previous similar message [ 1923.041168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1923.047868] Lustre: Skipped 4 previous similar messages [ 1928.169196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1928.183173] Lustre: Skipped 2 previous similar messages [ 1929.183120] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1929.366421] Lustre: server umount lustre-MDT0000 complete [ 1933.391495] LustreError: 66459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789449458 with bad export cookie 1939951813416798167 [ 1933.393576] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1933.405980] LustreError: 66459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1933.858449] Lustre: server umount lustre-MDT0001 complete [ 1937.588737] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1942.424113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1949.722290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1954.783314] Lustre: 63529:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789449463/real 1789449463] req@000000007413ffeb x1876373645863424/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789449479 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1954.814548] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1954.828693] Lustre: Skipped 1 previous similar message [ 1959.256271] LustreError: 66453:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1959.285519] LustreError: 66453:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 1959.390635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1959.482364] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1960.417859] Lustre: 63527:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789449468/real 1789449468] req@00000000545b744a x1876373645863552/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789449484 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1963.496356] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1963.499826] LustreError: 66454:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1963.650371] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1964.513977] LustreError: 68372:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1964.520309] Lustre: 63527:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789449473/real 1789449473] req@00000000e0aeaece x1876373645863808/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789449489 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1964.546442] LustreError: 68372:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1969.636726] Lustre: 63529:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789449478/real 1789449478] req@000000003ca0123e x1876373645863936/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789449494 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1969.662968] LustreError: 68373:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1969.728717] LustreError: 68373:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1972.276781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1972.893243] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1978.349761] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1978.604610] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 1987.520180] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1994.738954] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2003.424895] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2003.432701] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2003.456855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2008.013714] Lustre: server umount lustre-MDT0000 complete [ 2009.058531] LustreError: 66454:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2009.088323] LustreError: 66454:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2011.548071] LustreError: 66459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789449536 with bad export cookie 1939951813416800449 [ 2011.552543] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2011.572819] LustreError: 66459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2011.920347] Lustre: server umount lustre-MDT0001 complete [ 2018.643688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2028.334394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2030.559400] Lustre: 63528:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789449539/real 1789449539] req@00000000813f1373 x1876373645891328/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789449555 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2030.596989] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2030.608702] Lustre: Skipped 2 previous similar messages [ 2036.770035] LustreError: 63525:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000003da13560 x1876373645892352/t0(0) o250->MGC192.168.203.127@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2037.131661] LustreError: 69107:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2037.224411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2037.294091] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2041.494810] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2042.335287] Lustre: 63528:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789449551/real 1789449551] req@000000008d56a5fa x1876373645891968/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789449567 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2042.369498] Lustre: 63528:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 2049.400957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2053.833949] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2055.146603] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2059.235559] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2060.274593] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2065.975413] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2072.198229] Lustre: server umount lustre-MDT0000 complete [ 2074.600981] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2074.603156] LustreError: 71066:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2074.614064] Lustre: Skipped 2 previous similar messages [ 2074.635397] LustreError: 71066:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 2076.102481] LustreError: 64596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789449601 with bad export cookie 1939951813416801828 [ 2076.116867] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2076.134750] LustreError: 64596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2076.514773] Lustre: server umount lustre-MDT0001 complete [ 2083.055169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2086.549712] Lustre: DEBUG MARKER: files compared the same [ 2098.566358] Lustre: server umount lustre-OST0000 complete [ 2114.694840] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 2117.346221] Key type lgssc unregistered [ 2117.629954] LNet: 74337:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2117.645276] LNetError: 74337:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2117.668929] LNet: Removed LNI 192.168.203.127@tcp [ 2118.465345] Key type .llcrypt unregistered [ 2118.472577] Key type ._llcrypt unregistered [ 2143.304137] Key type ._llcrypt registered [ 2143.305855] Key type .llcrypt registered [ 2143.399576] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 2144.596585] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2144.665963] alg: No test for adler32 (adler32-zlib) [ 2145.777903] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 2146.068769] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 2147.823535] Key type lgssc registered [ 2148.897691] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2156.487511] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2156.522461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2157.960848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2162.371796] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug +malloc all [ 2167.265619] LustreError: 75752:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2167.302518] LustreError: 75752:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2170.380909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2170.812184] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2175.940646] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug +malloc all [ 2184.417717] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2191.171241] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2199.612268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2200.035643] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2202.090852] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2207.658938] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug +malloc all [ 2216.755692] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2227.686153] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2227.697046] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2227.719917] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2230.752241] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2230.753217] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2230.764601] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2230.800908] Lustre: Skipped 1 previous similar message [ 2233.625526] Lustre: server umount lustre-OST0000 complete [ 2240.997274] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2241.011444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2241.028921] Lustre: Skipped 1 previous similar message [ 2241.040349] Lustre: Skipped 1 previous similar message [ 2244.130041] LustreError: 75747:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2244.152823] LustreError: 75747:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2244.180941] Lustre: server umount lustre-MDT0000 complete [ 2248.033511] LustreError: 75733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789449773 with bad export cookie 3377290999149165341 [ 2248.057745] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2248.062366] LustreError: 75733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2248.414987] Lustre: server umount lustre-MDT0001 complete [ 2260.026204] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 2263.896704] Key type lgssc unregistered [ 2264.276475] LNet: 79287:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2264.285497] LNetError: 79287:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2264.311987] LNet: Removed LNI 192.168.203.127@tcp [ 2265.165367] Key type .llcrypt unregistered [ 2265.169163] Key type ._llcrypt unregistered [ 2291.724389] Key type ._llcrypt registered [ 2291.731472] Key type .llcrypt registered [ 2291.816757] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 2292.482561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2292.538577] alg: No test for adler32 (adler32-zlib) [ 2293.608740] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 2293.844801] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 2295.583164] Key type lgssc registered [ 2297.162805] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2306.924627] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2306.948686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2322.466577] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2327.583313] LustreError: 80676:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.127@tcp: failed processing log, type 4: rc = -110 [ 2354.207922] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2361.081976] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2368.578854] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2376.136449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2376.264726] Lustre: *** cfs_fail_loc=706, val=0*** [ 2376.279462] LustreError: 80009:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000086070164 x1876374132361472/t0(0) o250->MGC192.168.203.127@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2376.659315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2376.769174] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2381.346469] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2381.801944] LustreError: 80701:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2381.820802] LustreError: 80701:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2386.929416] LustreError: 80702:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2389.034838] LustreError: 81765:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2389.061750] LustreError: 81765:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2389.982359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2390.422478] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2394.689791] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2402.162954] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2407.541526] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2410.977187] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2410.978712] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2410.994590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2411.002275] Lustre: Skipped 1 previous similar message [ 2411.018312] Lustre: Skipped 1 previous similar message [ 2415.903529] Lustre: server umount lustre-OST0000 complete [ 2421.221096] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2421.227914] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2421.243534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2421.257131] Lustre: Skipped 1 previous similar message [ 2421.277303] Lustre: Skipped 1 previous similar message [ 2425.915364] Lustre: server umount lustre-MDT0000 complete [ 2426.338196] LustreError: 81771:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2426.353742] LustreError: 81771:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2429.340327] LustreError: 82508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789449954 with bad export cookie 16158606868479979210 [ 2429.346401] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2429.358626] LustreError: 82508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2429.641602] Lustre: server umount lustre-MDT0001 complete [ 2438.869873] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 2441.484512] Key type lgssc unregistered [ 2441.779212] LNet: 84294:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2441.790564] LNetError: 84294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2441.809573] LNet: Removed LNI 192.168.203.127@tcp [ 2442.855397] Key type .llcrypt unregistered [ 2442.857972] Key type ._llcrypt unregistered [ 2465.998175] Key type ._llcrypt registered [ 2466.001657] Key type .llcrypt registered [ 2466.117693] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 2467.148222] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2467.213928] alg: No test for adler32 (adler32-zlib) [ 2468.327569] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 2468.655557] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 2470.363306] Key type lgssc registered [ 2471.830516] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2481.244378] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2481.300678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2487.207628] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2496.700936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2497.365676] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2502.635171] LustreError: 86395:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2502.653661] LustreError: 86395:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2502.759276] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2507.751942] LustreError: 86398:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2507.794199] LustreError: 86398:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2512.039645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2512.367897] LustreError: 86395:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2512.395195] LustreError: 86395:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2512.403233] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2517.484132] LustreError: 86399:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2517.513911] LustreError: 86399:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2520.034193] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2522.594175] LustreError: 87143:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2522.621399] LustreError: 87143:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2528.602259] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2528.625297] Lustre: MGS: Client 1cbfdbba-613c-4ce8-99e1-b97196b4c868 (at 0@lo) reconnecting [ 2528.661269] LustreError: 86409:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2528.738702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 2528.745274] Lustre: Skipped 2 previous similar messages [ 2529.023206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2529.137522] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2533.626775] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2546.149068] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2546.167816] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2546.195050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2549.737693] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2549.738977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2549.741130] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2549.792064] Lustre: Skipped 1 previous similar message [ 2552.314447] Lustre: server umount lustre-OST0000 complete [ 2559.969092] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2559.975415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2559.976413] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2560.015757] Lustre: Skipped 1 previous similar message [ 2561.594849] Lustre: server umount lustre-MDT0000 complete [ 2565.117316] Lustre: server umount lustre-MDT0001 complete [ 2568.490853] Lustre: server umount MGS complete [ 2580.382549] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 01:28:24 (1789450104) [ 2608.830571] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [ 2618.120853] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 2651.267182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2655.058986] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2664.486987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2664.704115] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2664.766094] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2664.881113] Lustre: lustre-MDT0001: new disk, initializing [ 2664.987582] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2665.022929] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2665.034617] Lustre: 93564:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2665.045444] LustreError: 93564:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2669.473946] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2679.117889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2679.445898] Lustre: lustre-OST0000: new disk, initializing [ 2679.452368] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2679.468478] Lustre: 94332:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2679.568763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2685.420940] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2690.083042] Lustre: 94838:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000ab5fd39c x1876374315900416/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789450231 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2694.212874] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2700.462201] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2700.480303] Lustre: MGS: Client e3cda575-b28b-4e59-a6ba-8d202c1d4844 (at 0@lo) reconnecting [ 2700.507895] Lustre: 95377:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2700.523714] LustreError: 93549:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2700.571838] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2700.633788] Lustre: lustre-MDT0000: new disk, initializing [ 2700.786731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2700.824463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2700.958847] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2701.939917] Lustre: 95401:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000049b9c686 x1876374315906816/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1789450242 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2705.018744] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2707.060839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2707.075429] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2707.116542] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 2730.978446] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2730.988271] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2731.007871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2731.012206] Lustre: Skipped 1 previous similar message [ 2735.562830] Lustre: server umount lustre-OST0000 complete [ 2742.240493] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2742.244510] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2742.256649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2742.270899] Lustre: Skipped 2 previous similar messages [ 2742.276316] Lustre: Skipped 2 previous similar messages [ 2745.039736] Lustre: server umount lustre-MDT0000 complete [ 2747.369151] LustreError: 95715:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2747.388858] LustreError: 95715:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2754.830566] Lustre: server umount lustre-MDT0001 complete [ 2758.272935] Lustre: server umount MGS complete [ 2768.726318] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 01:31:32 (1789450292) [ 2783.521736] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 2785.987930] Key type lgssc unregistered [ 2786.323122] LNet: 97897:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2786.332871] LNetError: 97897:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2786.347570] LNet: Removed LNI 192.168.203.127@tcp [ 2787.111038] Key type .llcrypt unregistered [ 2787.113337] Key type ._llcrypt unregistered [ 2802.468717] Key type ._llcrypt registered [ 2802.470961] Key type .llcrypt registered [ 2802.572359] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 2803.686251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2803.719473] alg: No test for adler32 (adler32-zlib) [ 2804.758835] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 2804.996839] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 2806.640341] Key type lgssc registered [ 2807.719814] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2814.451813] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2814.467123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2815.829424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2819.440177] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2825.190917] LustreError: 99117:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2825.209332] LustreError: 99117:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2826.474432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2826.755532] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2830.975903] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2837.190084] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2841.755520] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2847.967755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2848.252397] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2853.773494] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2860.525652] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:33) [ 2861.067852] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2870.753060] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2870.761837] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2870.771476] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2870.784635] Lustre: Skipped 1 previous similar message [ 2874.627561] Lustre: server umount lustre-OST0000 complete [ 2877.920439] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2877.931293] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2877.943082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2877.950318] Lustre: Skipped 1 previous similar message [ 2880.992854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2880.994108] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2883.912292] Lustre: server umount lustre-MDT0000 complete [ 2886.113535] LustreError: 99117:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2886.130056] LustreError: 99117:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2887.099151] LustreError: 99849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789450412 with bad export cookie 14033576347850517302 [ 2887.102087] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2887.108084] LustreError: 99849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2887.498873] Lustre: server umount lustre-MDT0001 complete [ 2896.086059] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 2898.328313] Key type lgssc unregistered [ 2898.572353] LNet: 102645:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2898.580609] LNetError: 102645:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2898.603061] LNet: Removed LNI 192.168.203.127@tcp [ 2899.371127] Key type .llcrypt unregistered [ 2899.372783] Key type ._llcrypt unregistered [ 2912.651496] Key type ._llcrypt registered [ 2912.655923] Key type .llcrypt registered [ 2912.784395] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 2913.807182] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2913.919211] alg: No test for adler32 (adler32-zlib) [ 2914.995281] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 2915.312927] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 2917.031197] Key type lgssc registered [ 2918.074193] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2922.705606] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2922.705796] LustreError: 103705:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 2922.710020] Lustre: Skipped 1 previous similar message [ 2922.710040] LustreError: 103704:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 2922.730176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2922.730504] LustreError: 103705:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=5000 [ 2922.747391] LustreError: 103705:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2922.758125] LustreError: 103705:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2924.124691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2930.132238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2930.419111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2934.391345] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2939.089815] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing lsmod [ 2941.597971] LustreError: 104910:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 2941.610899] LustreError: 104911:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 2941.620730] LustreError: 104910:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4983 [ 2941.628200] LustreError: 104910:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2941.635872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2941.638338] LustreError: 104910:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2942.002279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2946.020361] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2946.027790] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2946.037902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2947.052637] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2947.054526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2947.075187] Lustre: Skipped 1 previous similar message [ 2947.084255] Lustre: Skipped 1 previous similar message [ 2950.509353] Lustre: server umount lustre-MDT0000 complete [ 2952.161077] LustreError: 103739:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2952.177946] LustreError: 103739:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2953.984267] LustreError: 103719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789450478 with bad export cookie 2252003794571532994 [ 2953.986947] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2953.995931] LustreError: 103719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2954.011207] LustreError: 105097:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2954.017278] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2954.054842] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2954.308743] Lustre: server umount lustre-MDT0001 complete [ 2957.912259] Lustre: server umount lustre-OST0000 complete [ 2965.017547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2965.369394] LustreError: 105974:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2965.435289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2969.053921] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2970.595259] LustreError: 105975:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2975.716644] LustreError: 105974:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2976.746343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2977.159688] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2981.364718] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 2988.305929] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2993.717874] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3001.031969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3001.602432] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3008.169280] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3010.863752] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:65) [ 3015.467331] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3023.331445] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3023.336740] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3023.350851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3026.407453] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3028.410242] Lustre: server umount lustre-OST0000 complete [ 3033.569084] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3033.582423] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3033.594294] Lustre: Skipped 1 previous similar message [ 3033.602596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3033.617447] Lustre: Skipped 2 previous similar messages [ 3038.009131] Lustre: server umount lustre-MDT0000 complete [ 3041.586416] LustreError: 105955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789450566 with bad export cookie 2252003794571533673 [ 3041.595053] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3041.902786] Lustre: server umount lustre-MDT0001 complete [ 3052.191269] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 3055.067961] Key type lgssc unregistered [ 3055.355760] LNet: 109508:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3055.363877] LNetError: 109508:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3055.378794] LNet: Removed LNI 192.168.203.127@tcp [ 3056.213910] Key type .llcrypt unregistered [ 3056.217836] Key type ._llcrypt unregistered [ 3076.756947] Key type ._llcrypt registered [ 3076.767566] Key type .llcrypt registered [ 3076.874735] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 3077.753218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3077.789507] alg: No test for adler32 (adler32-zlib) [ 3078.832214] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 3079.028463] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 3080.736072] Key type lgssc registered [ 3081.651476] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3088.299688] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3088.317763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3089.729231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3093.139250] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3099.106402] LustreError: 110899:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3099.128368] LustreError: 110899:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3099.975706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3100.241685] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3104.023938] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3110.891174] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3115.932657] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3122.178926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3122.628881] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3128.552355] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3134.972621] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:67 to 0x240000400:97) [ 3136.853981] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3142.531026] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3146.427030] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3155.435693] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3155.446498] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3155.473409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3155.483569] Lustre: Skipped 1 previous similar message [ 3157.862564] Lustre: server umount lustre-OST0000 complete [ 3165.664500] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3165.680673] Lustre: Skipped 1 previous similar message [ 3165.684131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3165.686577] Lustre: Skipped 1 previous similar message [ 3168.765577] Lustre: server umount lustre-MDT0000 complete [ 3170.784203] LustreError: 111952:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3170.801779] LustreError: 111952:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3172.308223] LustreError: 110879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789450697 with bad export cookie 10121351057363766123 [ 3172.313172] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3172.319464] LustreError: 110879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3172.553566] Lustre: server umount lustre-MDT0001 complete [ 3182.695580] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 3184.943663] Key type lgssc unregistered [ 3185.224962] LNet: 114536:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3185.235888] LNetError: 114536:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3185.256039] LNet: Removed LNI 192.168.203.127@tcp [ 3186.015127] Key type .llcrypt unregistered [ 3186.017898] Key type ._llcrypt unregistered [ 3200.949617] Key type ._llcrypt registered [ 3200.952180] Key type .llcrypt registered [ 3201.065956] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 3201.949605] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3202.091727] alg: No test for adler32 (adler32-zlib) [ 3203.177750] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 3203.423639] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 3205.103298] Key type lgssc registered [ 3206.260575] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3213.266382] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3213.286966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3214.715770] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3218.730964] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3224.049652] LustreError: 115747:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3224.072471] LustreError: 115747:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3226.962044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3227.285465] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3231.437946] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3238.051595] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3244.193382] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3252.229478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3252.510606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3259.718777] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 3260.124620] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3268.010809] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3283.427680] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3283.428681] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3283.462571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3283.468376] Lustre: Skipped 1 previous similar message [ 3286.953825] Lustre: server umount lustre-OST0000 complete [ 3293.664665] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3293.667118] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3293.667162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3293.674729] Lustre: Skipped 2 previous similar messages [ 3296.676040] Lustre: server umount lustre-MDT0000 complete [ 3298.784352] LustreError: 115748:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3298.800405] LustreError: 115748:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3299.617938] LustreError: 115729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789450824 with bad export cookie 4432611851818315968 [ 3299.631858] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3299.839054] Lustre: server umount lustre-MDT0001 complete [ 3308.097687] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 3310.374897] Key type lgssc unregistered [ 3310.616957] LNet: 119298:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3310.622269] LNetError: 119298:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3310.641868] LNet: Removed LNI 192.168.203.127@tcp [ 3311.296949] Key type .llcrypt unregistered [ 3311.303384] Key type ._llcrypt unregistered [ 3331.549102] Key type ._llcrypt registered [ 3331.551767] Key type .llcrypt registered [ 3331.666429] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 3333.023239] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3333.050070] alg: No test for adler32 (adler32-zlib) [ 3334.131897] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 3334.337561] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 3336.007211] Key type lgssc registered [ 3337.136758] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3344.496261] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3344.512835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3345.865879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3350.113308] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3355.107733] LustreError: 120690:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3355.135780] LustreError: 120690:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3358.602512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3358.995046] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3363.504098] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3370.739433] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3375.853922] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3382.099481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3382.515573] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3389.064496] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3390.710176] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:131 to 0x240000400:161) [ 3396.165968] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3401.926421] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3405.240087] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3405.248423] Lustre: Skipped 2 previous similar messages [ 3409.352795] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3409.356817] Lustre: Skipped 2 previous similar messages [ 3412.031381] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3412.071741] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3417.391570] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.27@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3417.399883] Lustre: Skipped 2 previous similar messages [ 3418.700857] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.27@tcp 192.168.0.[2,10]@tcp [ 3423.146360] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.27/24@tcp in log lustre-MDT0000 [ 3423.161298] Lustre: Skipped 2 previous similar messages [ 3428.446223] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.27/24@tcp [ 3428.451559] Lustre: Skipped 1 previous similar message [ 3434.981320] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3434.995560] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3435.030077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3436.019536] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3436.022396] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3436.025460] LustreError: Skipped 1 previous similar message [ 3436.041342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3440.186267] Lustre: server umount lustre-OST0000 complete [ 3446.240317] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3446.246049] LustreError: Skipped 4 previous similar messages [ 3446.252218] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3446.272510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3446.282591] Lustre: Skipped 2 previous similar messages [ 3450.000189] Lustre: server umount lustre-MDT0000 complete [ 3452.386563] LustreError: 121706:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3452.409596] LustreError: 121706:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3453.639313] LustreError: 124016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789450978 with bad export cookie 12851409544925413853 [ 3453.642772] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3453.653812] LustreError: 124016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3453.998532] Lustre: server umount lustre-MDT0001 complete [ 3463.848374] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 3466.630537] Key type lgssc unregistered [ 3466.908546] LNet: 125526:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3466.925824] LNetError: 125526:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3466.943785] LNet: Removed LNI 192.168.203.127@tcp [ 3467.799518] Key type .llcrypt unregistered [ 3467.802355] Key type ._llcrypt unregistered [ 3494.873839] Key type ._llcrypt registered [ 3494.876854] Key type .llcrypt registered [ 3494.977124] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 3496.185819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3496.248894] alg: No test for adler32 (adler32-zlib) [ 3497.571699] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 3497.886182] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 3499.551958] Key type lgssc registered [ 3500.721209] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3508.897915] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3508.926756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3510.435640] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3510.462063] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3510.465688] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.27/24@tcp [ 3515.441065] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3521.016022] LustreError: 127086:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3521.050898] LustreError: 127086:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3525.228609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3525.444695] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3525.476068] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.27/24@tcp [ 3525.534275] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3531.235960] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3540.264916] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3546.392761] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3553.059470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3553.371665] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3555.443376] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:169 to 0x240000400:193) [ 3559.405853] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3567.405531] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3579.884355] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3579.901952] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3579.927791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3581.408749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3581.411084] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3581.435673] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3585.896726] Lustre: server umount lustre-OST0000 complete [ 3591.649527] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3591.670817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3591.684899] Lustre: Skipped 2 previous similar messages [ 3596.772249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3596.784366] Lustre: Skipped 1 previous similar message [ 3597.139267] Lustre: server umount lustre-MDT0000 complete [ 3600.748984] LustreError: 127067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789451125 with bad export cookie 5464279676404911378 [ 3600.759218] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3600.776553] LustreError: 127067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3601.279071] Lustre: server umount lustre-MDT0001 complete [ 3612.491226] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 3615.523330] Key type lgssc unregistered [ 3615.838678] LNet: 130674:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3615.846398] LNetError: 130674:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3615.865119] LNet: Removed LNI 192.168.203.127@tcp [ 3616.680943] Key type .llcrypt unregistered [ 3616.695698] Key type ._llcrypt unregistered [ 3642.409616] Key type ._llcrypt registered [ 3642.413479] Key type .llcrypt registered [ 3642.499858] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [ 3656.135974] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 3657.182315] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3657.299192] alg: No test for adler32 (adler32-zlib) [ 3658.506663] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 3658.830501] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 3660.511636] Key type lgssc registered [ 3661.537736] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3695.331284] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3695.383498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3696.696602] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3696.740464] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3696.885977] Lustre: lustre-MDT0000: new disk, initializing [ 3696.981443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3697.010104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3702.056378] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3713.805175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3713.988140] Lustre: 134069:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3714.079107] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3714.095204] Lustre: Skipped 1 previous similar message [ 3714.286989] Lustre: lustre-MDT0001: new disk, initializing [ 3714.509831] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3714.558667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3714.577851] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3722.135448] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3732.482156] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3740.171815] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3749.820955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3750.171350] Lustre: lustre-OST0000: new disk, initializing [ 3750.176162] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3750.182644] Lustre: 136038:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3750.281433] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3755.641618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3755.663866] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3755.815634] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3758.675233] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3769.232770] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3775.845215] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3776.159985] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3781.489556] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3781.868458] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3786.724820] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3786.756909] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3786.756968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3786.771320] Lustre: Skipped 1 previous similar message [ 3790.017085] Lustre: server umount lustre-OST0000 complete [ 3796.448380] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3796.454903] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3796.466770] Lustre: Skipped 1 previous similar message [ 3796.474176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3802.082213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3802.090128] Lustre: Skipped 2 previous similar messages [ 3806.176733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3806.191834] Lustre: Skipped 1 previous similar message [ 3808.223158] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3808.440832] Lustre: server umount lustre-MDT0000 complete [ 3811.869095] LustreError: 134061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789451336 with bad export cookie 11823771882805635184 [ 3811.870457] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3811.890949] LustreError: 134061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3812.523703] Lustre: server umount lustre-MDT0001 complete [ 3821.469507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3821.957381] LustreError: 138256:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3821.988392] LustreError: 138256:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3822.086397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3826.718113] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3827.169618] LustreError: 138257:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3827.192078] LustreError: 138257:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3831.266286] LustreError: 138256:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3835.510700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3835.845864] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3840.758947] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3848.731516] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3853.969678] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3860.488650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3860.851296] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3866.836250] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3873.645209] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3877.913694] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3878.157635] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3882.246310] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3882.464204] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3891.972308] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3892.187171] Lustre: lustre-OST0001: new disk, initializing [ 3892.194579] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3892.202444] Lustre: 141624:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3892.302881] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3897.908946] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 3901.005672] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3901.023710] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3901.150212] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3906.471637] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3906.698939] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3912.394609] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3912.581747] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3918.414515] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9dbfe0b34800.ost_server_uuid 50 [ 3919.768668] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9dbfe0b34800.ost_server_uuid in IDLE state after 0 sec [ 3932.145796] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3932.151709] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3932.156635] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3932.164721] Lustre: Skipped 2 previous similar messages [ 3932.183921] Lustre: Skipped 1 previous similar message [ 3932.514097] Lustre: server umount lustre-OST0001 complete [ 3936.742098] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3936.749723] LustreError: Skipped 1 previous similar message [ 3936.760538] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3937.251396] LustreError: 141993:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3937.269950] LustreError: 141993:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3942.372878] Lustre: server umount lustre-OST0000 complete [ 3947.488430] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3947.508883] Lustre: Skipped 1 previous similar message [ 3947.517246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3947.522710] Lustre: Skipped 4 previous similar messages [ 3952.843497] Lustre: server umount lustre-MDT0000 complete [ 3957.442625] LustreError: 142928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789451482 with bad export cookie 11823771882805636066 [ 3957.447646] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3957.451788] LustreError: 142928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3958.008960] Lustre: server umount lustre-MDT0001 complete [ 3978.935097] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 3982.175612] Key type lgssc unregistered [ 3982.531189] LNet: 144629:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3982.544332] LNetError: 144629:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3982.562672] LNet: Removed LNI 192.168.203.127@tcp [ 3983.343190] Key type .llcrypt unregistered [ 3983.345180] Key type ._llcrypt unregistered [ 4032.846959] Key type ._llcrypt registered [ 4032.851290] Key type .llcrypt registered [ 4033.022516] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [ 4057.395630] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 4058.302264] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4058.410291] alg: No test for adler32 (adler32-zlib) [ 4059.664910] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 4060.038651] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 4061.760181] Key type lgssc registered [ 4063.562063] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4101.744438] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4101.764640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4103.033314] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4103.081858] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4103.232740] Lustre: lustre-MDT0000: new disk, initializing [ 4103.310817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4103.345063] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4108.050875] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4117.469793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4117.595422] Lustre: 148782:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4117.663778] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4117.668226] Lustre: Skipped 1 previous similar message [ 4117.727986] Lustre: lustre-MDT0001: new disk, initializing [ 4117.804422] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4117.856946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4117.881091] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4123.087937] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4132.134133] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4138.436889] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4146.067545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4146.321782] Lustre: lustre-OST0000: new disk, initializing [ 4146.326553] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4146.335823] Lustre: 150751:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4146.428409] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4152.262128] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4155.979958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4155.993346] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4156.152965] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4161.841232] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4165.879242] Lustre: Failing over lustre-OST0000 [ 4166.050231] Lustre: server umount lustre-OST0000 complete [ 4183.007455] Lustre: 147004:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789451691/real 1789451691] req@00000000293ab9f7 x1876375985226624/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789451707 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4183.041718] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4184.097164] Lustre: 147007:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789451693/real 1789451693] req@00000000eb0a24fa x1876375985227392/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1789451709 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4184.134671] Lustre: 147007:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4185.076764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4185.705232] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4185.723739] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4187.680232] Lustre: 147004:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789451696/real 1789451696] req@00000000ae480281 x1876375985228544/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789451712 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4191.915519] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4192.867529] Lustre: 147005:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789451701/real 1789451701] req@00000000bc50b1b0 x1876375985229568/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789451717 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4192.888624] Lustre: 147005:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4193.833997] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4194.129490] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4194.131154] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4195.779345] Lustre: Failing over lustre-MDT0000 [ 4196.168098] Lustre: server umount lustre-MDT0000 complete [ 4197.857205] Lustre: 147005:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789451706/real 1789451706] req@00000000a2c9d9a0 x1876375985230720/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789451722 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4197.888202] Lustre: 147005:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4198.887216] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4198.908675] Lustre: Skipped 1 previous similar message [ 4204.003910] LustreError: 148788:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4204.036778] LustreError: 148788:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 4209.120606] LustreError: 149650:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4209.139929] LustreError: 149650:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4214.239264] Lustre: 147007:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789451723/real 1789451723] req@00000000526a9df7 x1876375985240320/t0(0) o400->MGC192.168.203.127@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789451739 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4214.242722] LustreError: 148793:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4214.267410] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4214.314055] LustreError: 148793:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4214.779065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4219.360919] LustreError: 149650:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4219.383499] LustreError: 149650:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 4224.480149] LustreError: 147003:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000c010615e x1876375985245184/t0(0) o250->MGC192.168.203.127@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4224.501644] LustreError: 148793:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4224.516334] LustreError: 148793:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4224.749071] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4224.810754] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4228.759595] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4229.917204] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4230.137362] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4230.145231] Lustre: Skipped 1 previous similar message [ 4230.170296] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4230.219260] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4240.360702] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4240.366390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4240.389718] Lustre: Skipped 3 previous similar messages [ 4240.398870] Lustre: Skipped 1 previous similar message [ 4243.502893] Lustre: server umount lustre-OST0000 complete [ 4250.593814] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4250.597418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4250.613906] Lustre: Skipped 1 previous similar message [ 4250.627790] Lustre: Skipped 1 previous similar message [ 4255.716553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4255.722511] Lustre: Skipped 1 previous similar message [ 4260.832971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4260.839365] Lustre: Skipped 1 previous similar message [ 4261.343123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4261.463186] Lustre: server umount lustre-MDT0000 complete [ 4265.042233] LustreError: 150750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789451790 with bad export cookie 18207818280488852362 [ 4265.045951] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4265.055523] LustreError: 150750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4265.460320] Lustre: server umount lustre-MDT0001 complete [ 4275.328347] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 4277.968454] Key type lgssc unregistered [ 4278.210512] LNet: 154408:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4278.230200] LNetError: 154408:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4279.270076] LNet: Removed LNI 192.168.203.127@tcp [ 4280.344404] Key type .llcrypt unregistered [ 4280.348299] Key type ._llcrypt unregistered [ 4315.540843] Key type ._llcrypt registered [ 4315.543568] Key type .llcrypt registered [ 4315.637838] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 4316.680157] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4316.735304] alg: No test for adler32 (adler32-zlib) [ 4317.826764] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 4318.070614] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 4319.735202] Key type lgssc registered [ 4320.848793] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4328.290720] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4328.319615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4329.602579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4333.584875] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4339.173227] LustreError: 155923:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4339.198812] LustreError: 155923:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4342.873962] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4344.294632] LustreError: 155922:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4346.586899] LustreError: 155951:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4346.602413] LustreError: 155951:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4346.611951] LustreError: 155951:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 0, failed: rc = -5 [ 4346.943256] Lustre: server umount lustre-MDT0000 complete [ 4361.862224] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [ 4373.538730] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 4411.390921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4411.783135] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4411.862074] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4411.999377] Lustre: lustre-MDT0000: new disk, initializing [ 4412.194367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4412.238563] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4418.036951] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4431.361536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4431.602424] Lustre: 159256:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4431.684500] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4431.690185] Lustre: Skipped 1 previous similar message [ 4431.824459] Lustre: lustre-MDT0001: new disk, initializing [ 4431.986324] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4432.021697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4432.042464] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4437.448736] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4447.132476] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4452.466577] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4460.422301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4460.778617] Lustre: lustre-OST0000: new disk, initializing [ 4460.781700] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4460.786955] Lustre: 161228:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4460.856122] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4466.908121] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4469.865773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4469.881214] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4470.098203] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4476.943749] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4481.377078] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4482.858398] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 02:00:06 (1789452006) [ 4710.879794] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4710.880987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4710.891614] Lustre: Skipped 2 previous similar messages [ 4710.899981] Lustre: Skipped 1 previous similar message [ 4716.013126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4716.032843] Lustre: Skipped 3 previous similar messages [ 4716.247748] Lustre: server umount lustre-MDT0000 complete [ 4720.093379] LustreError: 159247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789452245 with bad export cookie 13696331611019395140 [ 4720.093981] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4720.133195] LustreError: 159247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4721.122050] LustreError: 159262:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4721.124187] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4721.172159] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4725.151771] LustreError: 159264:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4725.152662] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4726.653541] Lustre: server umount lustre-MDT0001 complete [ 4737.827627] Lustre: server umount lustre-OST0000 complete [ 4748.805825] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [ 4759.527579] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 4793.595338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4793.789645] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4793.821769] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4793.882754] Lustre: lustre-MDT0000: new disk, initializing [ 4793.933139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4793.951037] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4798.741865] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4808.167598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4808.305548] Lustre: 165897:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4808.354986] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4808.358465] Lustre: Skipped 1 previous similar message [ 4808.417481] Lustre: lustre-MDT0001: new disk, initializing [ 4808.481648] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4808.518954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4808.529660] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4812.520435] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4822.138661] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4828.042254] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4833.911620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4834.169410] Lustre: lustre-OST0000: new disk, initializing [ 4834.174207] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4834.183235] Lustre: 167868:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4834.288467] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4840.439251] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4843.096698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4843.117649] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4843.191831] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4849.631283] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4854.738895] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4855.005543] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4860.041776] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4860.279142] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4863.971164] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4863.980526] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4864.003276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4864.026896] Lustre: Skipped 1 previous similar message [ 4868.420332] Lustre: server umount lustre-OST0000 complete [ 4874.214789] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4874.219601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4874.245295] Lustre: Skipped 2 previous similar messages [ 4874.255845] Lustre: Skipped 1 previous similar message [ 4878.440869] Lustre: server umount lustre-MDT0000 complete [ 4879.333900] LustreError: 166904:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4879.372478] LustreError: 166904:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4882.709790] LustreError: 165890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789452407 with bad export cookie 13696331611019553970 [ 4882.716474] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4882.722910] LustreError: 165890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4883.058244] Lustre: server umount lustre-MDT0001 complete [ 4892.906638] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:06:56 (1789452416) [ 4912.613389] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [ 4926.614289] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 4961.756407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4962.070517] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4962.108253] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4962.208523] Lustre: lustre-MDT0000: new disk, initializing [ 4962.314689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4962.338601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4966.962858] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4977.289166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4977.457422] Lustre: 172625:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4977.735049] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4982.549583] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 4991.595547] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4997.496222] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5004.967026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5005.237510] Lustre: lustre-OST0000: new disk, initializing [ 5005.241242] Lustre: Skipped 1 previous similar message [ 5005.246952] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5005.251654] Lustre: Skipped 2 previous similar messages [ 5005.256958] Lustre: 174600:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5012.033722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5012.055090] Lustre: Skipped 1 previous similar message [ 5012.059830] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5012.210895] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5013.353217] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5024.479883] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5032.931903] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5032.932727] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5032.944865] LustreError: Skipped 1 previous similar message [ 5032.965809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5032.977922] Lustre: Skipped 1 previous similar message [ 5038.497800] Lustre: server umount lustre-OST0000 complete [ 5043.185037] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5043.198399] Lustre: Skipped 2 previous similar messages [ 5049.128852] Lustre: server umount lustre-MDT0000 complete [ 5052.288909] LustreError: 172616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789452577 with bad export cookie 13696331611019554845 [ 5052.301574] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5052.552812] Lustre: server umount lustre-MDT0001 complete [ 5060.166584] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:09:43 (1789452583) [ 5074.885986] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [ 5085.149504] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 5126.864145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5127.055386] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5127.084876] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5127.152338] Lustre: lustre-MDT0000: new disk, initializing [ 5127.234330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5127.244384] Lustre: Skipped 2 previous similar messages [ 5127.271046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5131.681975] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5141.952514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5142.125968] Lustre: 178926:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5142.532753] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5147.140340] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5156.831931] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5162.630860] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5168.760589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5168.997084] Lustre: 180902:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5174.464757] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5175.088436] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5185.589265] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5195.233936] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5195.234673] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5195.254497] LustreError: Skipped 1 previous similar message [ 5195.255474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5195.255478] Lustre: Skipped 6 previous similar messages [ 5195.325733] Lustre: Skipped 1 previous similar message [ 5201.249167] Lustre: server umount lustre-OST0000 complete [ 5209.059618] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5220.319618] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5220.492009] Lustre: server umount lustre-MDT0000 complete [ 5220.834231] LustreError: 178936:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5220.861948] LustreError: 178936:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5225.566776] LustreError: 178917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789452750 with bad export cookie 13696331611019555937 [ 5225.584211] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5225.588522] LustreError: 178917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 5226.135442] Lustre: server umount lustre-MDT0001 complete [ 5239.530443] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 5242.474627] Key type lgssc unregistered [ 5242.870878] LNet: 182761:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5242.885195] LNetError: 182761:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5242.908246] LNet: Removed LNI 192.168.203.127@tcp [ 5244.026357] Key type .llcrypt unregistered [ 5244.028279] Key type ._llcrypt unregistered [ 5275.955033] Key type ._llcrypt registered [ 5275.956788] Key type .llcrypt registered [ 5276.087555] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 5277.341881] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5277.539162] alg: No test for adler32 (adler32-zlib) [ 5278.729398] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 5278.979143] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 5280.631218] Key type lgssc registered [ 5281.842230] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5289.883154] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5289.906759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5291.436833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5296.644794] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5301.736307] LustreError: 184176:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5301.763538] LustreError: 184176:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5305.991663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5306.598618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5313.661693] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5323.877382] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5331.184096] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5339.642248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5340.153493] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5347.230161] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5355.549798] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5361.650824] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5382.135992] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5382.146891] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5382.154472] LustreError: Skipped 1 previous similar message [ 5382.163527] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5382.179149] Lustre: Skipped 1 previous similar message [ 5382.193360] Lustre: Skipped 1 previous similar message [ 5387.244478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5387.252317] Lustre: Skipped 1 previous similar message [ 5388.108860] Lustre: server umount lustre-OST0000 complete [ 5392.353470] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5392.357484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5392.367830] Lustre: Skipped 1 previous similar message [ 5392.381104] Lustre: Skipped 1 previous similar message [ 5397.472747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5397.478708] Lustre: Skipped 1 previous similar message [ 5397.920642] Lustre: server umount lustre-MDT0000 complete [ 5401.729217] LustreError: 184158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789452926 with bad export cookie 16128130095263246913 [ 5401.742067] LustreError: 184158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5401.745994] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5402.313702] Lustre: server umount lustre-MDT0001 complete [ 5413.230346] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 5416.041384] Key type lgssc unregistered [ 5416.343911] LNet: 187737:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5416.351350] LNetError: 187737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5417.383320] LNet: Removed LNI 192.168.203.127@tcp [ 5418.305461] Key type .llcrypt unregistered [ 5418.307854] Key type ._llcrypt unregistered [ 5440.203143] Key type ._llcrypt registered [ 5440.205020] Key type .llcrypt registered [ 5440.328685] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 5441.611751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5441.639249] alg: No test for adler32 (adler32-zlib) [ 5442.676886] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 5442.892289] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 5444.512936] Key type lgssc registered [ 5445.547750] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5454.020991] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5454.054800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5455.531701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5460.013510] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5465.058633] LustreError: 189149:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5465.086307] LustreError: 189149:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5468.271554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5468.611725] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5473.021618] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5479.945414] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5485.397872] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5491.035934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5491.230584] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5500.442083] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5500.495097] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5509.412814] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5515.749627] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5515.755374] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5515.757086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5515.781913] Lustre: Skipped 1 previous similar message [ 5516.611085] Lustre: lustre-OST0000: Not available for connect from 192.168.203.27@tcp (stopping) [ 5516.619721] Lustre: Skipped 1 previous similar message [ 5520.875027] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5520.893164] Lustre: Skipped 1 previous similar message [ 5521.540029] Lustre: server umount lustre-OST0000 complete [ 5528.797779] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9dbfc7675800.ost_server_uuid 50 [ 5530.239826] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5545.441620] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5545.451880] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5545.461845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5546.466445] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5549.143497] Lustre: server umount lustre-MDT0000 complete [ 5551.598453] LustreError: 189148:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5551.624205] LustreError: 189148:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5552.544346] LustreError: 189130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789453077 with bad export cookie 14884040772766066605 [ 5552.547356] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5552.555370] LustreError: 189130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5552.970997] Lustre: server umount lustre-MDT0001 complete [ 5560.063068] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 02:18:03 (1789453083) [ 5566.232521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5566.642531] LustreError: 193065:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5566.725244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5571.330208] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5572.069657] LustreError: 193066:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5576.163329] LustreError: 193065:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5579.949361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5580.388495] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5585.075600] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5594.576694] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5600.569190] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5607.300708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5607.815119] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5611.911910] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5614.442436] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5621.707851] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5627.526204] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5627.712264] Lustre: lustre-OST0001: new disk, initializing [ 5627.715499] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5627.722825] Lustre: 195908:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5627.798329] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5633.103187] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5634.121889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5634.138316] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5634.278920] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5641.699299] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5644.774793] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5644.781525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5644.791717] Lustre: Skipped 1 previous similar message [ 5644.813145] Lustre: Skipped 3 previous similar messages [ 5650.426650] Lustre: server umount lustre-OST0000 complete [ 5652.752618] LustreError: 194888:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5655.862867] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5656.067765] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5660.306824] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5660.533178] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5670.372063] LustreError: 194888:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5670.398132] LustreError: 194888:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 5675.499707] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5675.506750] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5675.511182] LustreError: Skipped 1 previous similar message [ 5675.514823] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5675.535655] Lustre: Skipped 1 previous similar message [ 5675.537861] Lustre: Skipped 3 previous similar messages [ 5680.025803] Lustre: server umount lustre-OST0001 complete [ 5685.734832] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5685.749829] Lustre: Skipped 1 previous similar message [ 5690.037548] Lustre: server umount lustre-MDT0000 complete [ 5693.385312] LustreError: 195907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789453218 with bad export cookie 14884040772766067676 [ 5693.389883] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5693.394106] LustreError: 195907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5693.791704] Lustre: server umount lustre-MDT0001 complete [ 5717.002063] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 02:20:40 (1789453240) [ 5721.919389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5722.134333] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5722.168220] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5722.377362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5726.702057] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5734.437918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5734.523864] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5734.559554] Lustre: 199351:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5738.610133] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5745.182618] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5750.524958] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5756.478931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5756.614228] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5763.077235] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5766.749413] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 5770.343651] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5776.707339] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5776.753692] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5782.528262] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5784.341095] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5789.475933] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5792.224433] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5792.234181] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5792.247141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5792.255979] Lustre: Skipped 3 previous similar messages [ 5798.030866] Lustre: server umount lustre-OST0000 complete [ 5799.919406] LustreError: 201198:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5799.937349] LustreError: 201198:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 5815.272500] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5815.283478] LustreError: Skipped 2 previous similar messages [ 5819.486619] Lustre: server umount lustre-OST0001 complete [ 5825.514595] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5825.521514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5825.532299] Lustre: Skipped 4 previous similar messages [ 5825.544414] Lustre: Skipped 7 previous similar messages [ 5831.453614] Lustre: server umount lustre-MDT0000 complete [ 5835.347394] LustreError: 199341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789453360 with bad export cookie 14884040772766069083 [ 5835.351409] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5835.360133] LustreError: 199341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5835.768948] Lustre: server umount lustre-MDT0001 complete [ 5856.587754] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 02:23:00 (1789453380) [ 5870.128569] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [ 5878.352550] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 5909.919626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5910.093897] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5910.121389] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5910.179583] Lustre: lustre-MDT0000: new disk, initializing [ 5910.272444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5910.280123] Lustre: Skipped 3 previous similar messages [ 5910.291316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5914.523228] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5925.741348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5925.902457] Lustre: 207438:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5925.974472] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5925.980509] Lustre: Skipped 1 previous similar message [ 5926.085846] Lustre: lustre-MDT0001: new disk, initializing [ 5926.309205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5926.328589] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5930.895501] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5939.572090] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5945.580857] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5951.985070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5952.267901] Lustre: lustre-OST0000: new disk, initializing [ 5952.272415] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5952.285319] Lustre: 209408:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5958.357413] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 5962.297444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5962.307226] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5962.452348] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5966.245316] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5970.430485] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5970.690669] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5974.625128] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5974.838841] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5978.081353] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5978.085509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5978.096320] Lustre: Skipped 1 previous similar message [ 5978.096902] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5978.103046] Lustre: Skipped 1 previous similar message [ 5982.753905] Lustre: server umount lustre-OST0000 complete [ 5992.327158] Lustre: server umount lustre-MDT0000 complete [ 5993.443975] LustreError: 209003:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5993.472259] LustreError: 209003:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 12 previous similar messages [ 5996.481820] LustreError: 207431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789453521 with bad export cookie 14884040772766070868 [ 5996.485088] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5996.490319] LustreError: 207431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5996.769852] Lustre: server umount lustre-MDT0001 complete [ 6005.002535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6009.062544] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6016.467520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6020.330378] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6026.206825] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6031.246587] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6037.040513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6041.656819] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6047.631856] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6051.287406] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6051.452694] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6054.898271] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6055.104336] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6056.930191] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6056.936759] LustreError: Skipped 1 previous similar message [ 6063.091535] Lustre: server umount lustre-OST0000 complete [ 6068.143220] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6072.248517] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6100.358872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6100.656543] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6100.663366] Lustre: Skipped 5 previous similar messages [ 6101.675848] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6101.692910] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6105.511298] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6111.913062] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6111.926937] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6112.429597] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6119.349976] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 5 sec [ 6127.589093] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6127.592070] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6127.594764] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6127.594771] Lustre: Skipped 6 previous similar messages [ 6127.602309] Lustre: Skipped 5 previous similar messages [ 6127.639530] LustreError: Skipped 2 previous similar messages [ 6129.103172] Lustre: server umount lustre-OST0000 complete [ 6138.530885] Lustre: server umount lustre-MDT0000 complete [ 6141.433857] LustreError: 211607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789453666 with bad export cookie 14884040772766071750 [ 6141.441137] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6141.443643] LustreError: 211607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6141.694948] Lustre: server umount lustre-MDT0001 complete [ 6149.333586] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 02:27:52 (1789453672) [ 6155.476161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6155.893357] LustreError: 217590:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6155.916528] LustreError: 217590:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 6159.689534] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6168.074228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6172.077913] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6178.714057] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6182.853315] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6188.184456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6194.374089] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6201.244863] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6206.023515] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6206.165411] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6210.863838] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6211.180307] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6218.891511] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6219.127914] Lustre: lustre-OST0001: new disk, initializing [ 6219.131774] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6219.139338] Lustre: 220959:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6225.096464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6225.114481] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6225.326811] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6226.624411] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6236.312462] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6240.781230] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6241.114885] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6246.703025] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6246.917921] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6250.980423] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6255.377263] Lustre: server umount lustre-OST0001 complete [ 6260.893526] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6261.155271] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6265.211738] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6265.427577] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6293.468303] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6298.846400] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6299.111726] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6299.128366] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6299.132741] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6305.224428] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6306.759777] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6314.486436] Lustre: server umount lustre-OST0001 complete [ 6345.183139] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6345.345953] Lustre: server umount lustre-OST0000 complete [ 6354.992537] Lustre: server umount lustre-MDT0000 complete [ 6361.712542] LustreError: 217572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789453886 with bad export cookie 14884040772766072716 [ 6361.721745] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6361.727886] LustreError: 217572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6368.211067] Lustre: server umount lustre-MDT0001 complete [ 6390.127516] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 02:31:53 (1789453913) [ 6396.528638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6396.847312] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6396.891852] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6397.068733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6397.075595] Lustre: Skipped 5 previous similar messages [ 6401.416678] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6412.031705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6412.261621] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6412.291271] Lustre: 226206:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6417.954404] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6426.065441] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6432.495433] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6439.501569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6439.749464] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6444.797584] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6450.386482] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6458.460713] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6458.530215] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6463.358749] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6468.989846] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6470.338282] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6474.112632] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6474.354323] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6478.024367] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6478.239580] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6482.538467] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9dbfc31e3000.ost_server_uuid 50 [ 6483.788678] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9dbfc31e3000.ost_server_uuid in IDLE state after 0 sec [ 6484.551198] Lustre: Permanently deactivating lustre-OST0001 [ 6484.567555] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6487.891073] Lustre: Permanently reactivating lustre-OST0001 [ 6495.713459] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6495.721623] LustreError: Skipped 5 previous similar messages [ 6495.727061] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6495.741063] Lustre: Skipped 10 previous similar messages [ 6495.745474] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6495.749993] Lustre: Skipped 20 previous similar messages [ 6497.288158] Lustre: server umount lustre-OST0001 complete [ 6501.349797] LustreError: 228054:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6501.394167] LustreError: 228054:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 44 previous similar messages [ 6506.916984] Lustre: server umount lustre-OST0000 complete [ 6516.359638] Lustre: server umount lustre-MDT0000 complete [ 6519.572720] LustreError: 226199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789454044 with bad export cookie 14884040772766074004 [ 6519.579116] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6519.588361] LustreError: 226199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6519.850843] Lustre: server umount lustre-MDT0001 complete [ 6539.399996] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 02:34:23 (1789454063) [ 6545.998610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6546.239143] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6546.289060] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6546.296613] Lustre: Skipped 5 previous similar messages [ 6549.917600] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6556.588568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6556.688442] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6556.721955] Lustre: 232895:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6561.128592] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6567.659663] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6572.965555] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6578.623528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6578.801510] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6578.818669] Lustre: Permanently deactivating lustre-OST0000 [ 6578.828106] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6584.104645] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6585.385442] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6590.422241] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6595.886454] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6595.942622] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6601.573101] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6604.516995] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6608.064596] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6609.424052] Lustre: Permanently reactivating lustre-OST0000 [ 6609.443195] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6609.448210] Lustre: Skipped 2 previous similar messages [ 6609.754622] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6609.763667] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6609.772268] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6609.775064] Lustre: Skipped 1 previous similar message [ 6609.816315] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6609.839846] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6626.568826] Lustre: server umount lustre-OST0001 complete [ 6644.191373] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6644.350394] Lustre: server umount lustre-OST0000 complete [ 6653.680735] Lustre: server umount lustre-MDT0000 complete [ 6657.057236] LustreError: 232887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789454182 with bad export cookie 14884040772766075656 [ 6657.058891] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6657.068508] LustreError: 232887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6657.386499] Lustre: server umount lustre-MDT0001 complete [ 6665.438398] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 6667.492691] Key type lgssc unregistered [ 6667.718340] LNet: 237774:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6667.724670] LNetError: 237774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6667.741317] LNet: Removed LNI 192.168.203.127@tcp [ 6668.361451] Key type .llcrypt unregistered [ 6668.363216] Key type ._llcrypt unregistered [ 6690.126984] Key type ._llcrypt registered [ 6690.129780] Key type .llcrypt registered [ 6690.229170] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 6691.119393] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6691.141902] alg: No test for adler32 (adler32-zlib) [ 6692.184640] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 6692.390159] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 6694.047257] Key type lgssc registered [ 6694.913341] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6703.631095] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6703.655338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6705.336182] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6709.509500] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6715.874730] LustreError: 239237:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6715.889773] LustreError: 239237:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6716.885090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6716.967201] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6717.021337] Lustre: 239227:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6717.037952] Lustre: Permanently deactivating lustre-MDT0001 [ 6717.062124] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6717.350415] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6721.389599] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6722.049944] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6722.055693] LustreError: 239269:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6722.067824] LustreError: 239269:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6722.076041] LustreError: 239269:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 1, failed: rc = -108 [ 6727.106724] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6731.358860] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6738.218482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6738.542298] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6742.637741] LustreError: 241080:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6742.657986] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 6743.826275] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6748.146620] LustreError: 241080:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6750.203273] LustreError: 241081:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6750.217382] LustreError: 241081:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6750.223760] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6751.158775] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6755.304523] LustreError: 241080:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6755.325836] LustreError: 241080:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6757.202991] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6757.351959] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6762.491481] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6762.506248] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6763.447766] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6769.914246] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6772.658101] Lustre: Permanently reactivating lustre-MDT0001 [ 6772.681355] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6772.689816] Lustre: Skipped 2 previous similar messages [ 6778.683060] Lustre: lustre-MDT0001: Client 0b1a6752-df3b-497d-8b1c-805697980f03 (at 192.168.203.27@tcp) reconnecting [ 6784.983586] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6786.453340] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6791.790068] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6792.079599] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6797.030835] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6797.295874] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6799.839741] Lustre: Permanently deactivating lustre-MDT0001 [ 6799.871899] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6799.879799] Lustre: Skipped 2 previous similar messages [ 6807.623496] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6819.814237] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6819.833511] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6819.856394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6821.859268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6821.860551] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6821.868444] Lustre: Skipped 2 previous similar messages [ 6821.883230] Lustre: Skipped 2 previous similar messages [ 6823.501405] Lustre: server umount lustre-MDT0000 complete [ 6826.623103] LustreError: 239219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789454351 with bad export cookie 17219760395073383721 [ 6826.634352] LustreError: 239219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6826.635289] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6826.945462] Lustre: server umount lustre-MDT0001 complete [ 6835.842205] Lustre: server umount lustre-OST0000 complete [ 6845.948320] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 02:39:29 (1789454369) [ 6856.661903] Lustre: server umount lustre-OST0001 complete [ 6863.666959] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [ 6872.825573] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 6902.479892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6902.620148] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6902.628046] Lustre: Skipped 2 previous similar messages [ 6902.651594] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6902.718706] Lustre: lustre-MDT0000: new disk, initializing [ 6902.777203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6902.798780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6906.206239] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6915.289056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6915.428216] Lustre: 247541:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6915.502818] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6915.509201] Lustre: Skipped 1 previous similar message [ 6915.619399] Lustre: lustre-MDT0001: new disk, initializing [ 6915.738789] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6915.780540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6915.792203] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6919.597131] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6926.999593] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6931.744644] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6937.761064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6937.973228] Lustre: lustre-OST0000: new disk, initializing [ 6937.978881] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6937.985478] Lustre: 249506:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6938.063789] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6943.514694] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6944.821363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6944.849852] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6944.949204] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6951.566357] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6953.545742] LustreError: 248670:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6962.492800] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6962.701497] Lustre: lustre-OST0001: new disk, initializing [ 6962.715322] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6962.731723] Lustre: 250622:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6963.607519] LustreError: 248670:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6963.623074] LustreError: 250630:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6969.434273] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 6971.551126] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6971.574887] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6971.700372] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6973.729443] LustreError: 250630:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6980.294796] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6981.886433] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6987.243449] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6987.249056] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6987.271168] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6987.272692] Lustre: Skipped 1 previous similar message [ 6990.049559] Lustre: server umount lustre-OST0001 complete [ 6992.360205] LustreError: 249491:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6992.380034] LustreError: 249491:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7006.689783] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7006.695405] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7006.696948] LustreError: Skipped 1 previous similar message [ 7006.699931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7006.709297] Lustre: Skipped 1 previous similar message [ 7006.726080] Lustre: Skipped 2 previous similar messages [ 7012.770751] Lustre: server umount lustre-OST0000 complete [ 7017.952676] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7017.980924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7017.990703] Lustre: Skipped 3 previous similar messages [ 7022.486768] Lustre: server umount lustre-MDT0000 complete [ 7023.073198] LustreError: 247545:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7023.087819] LustreError: 247545:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 7025.554377] LustreError: 247531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789454550 with bad export cookie 17219760395073386535 [ 7025.559352] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7025.568308] LustreError: 247531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7032.113591] Lustre: server umount lustre-MDT0001 complete [ 7041.064455] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 7043.585084] Key type lgssc unregistered [ 7043.973624] LNet: 252655:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7043.982482] LNetError: 252655:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7044.000700] LNet: Removed LNI 192.168.203.127@tcp [ 7044.711264] Key type .llcrypt unregistered [ 7044.715782] Key type ._llcrypt unregistered [ 7083.872027] Key type ._llcrypt registered [ 7083.874076] Key type .llcrypt registered [ 7083.936378] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 7084.952662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7085.007667] alg: No test for adler32 (adler32-zlib) [ 7086.186735] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 7086.524265] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 7088.191490] Key type lgssc registered [ 7089.325440] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7097.948722] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7097.962935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7099.220735] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7099.262743] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7099.370581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7104.173203] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 7112.336096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7112.403305] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7112.424373] Lustre: 254862:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7112.665469] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7117.479694] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 7125.801729] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7131.674873] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7138.982314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7139.195994] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7139.481076] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7146.739110] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 7148.786762] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7155.048128] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7210.468322] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7210.471135] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7210.496523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7210.517733] Lustre: Skipped 1 previous similar message [ 7211.972535] Lustre: server umount lustre-OST0000 complete [ 7216.243226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7227.619147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7228.055209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7230.123478] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7230.139979] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7230.156739] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7230.172887] Lustre: Skipped 1 previous similar message [ 7233.382639] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 7239.672277] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7281.647631] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7281.654840] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7281.660726] LustreError: Skipped 1 previous similar message [ 7281.661961] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7281.710305] Lustre: Skipped 2 previous similar messages [ 7285.761247] Lustre: server umount lustre-OST0000 complete [ 7291.878122] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7291.879942] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7291.881336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7291.881343] Lustre: Skipped 1 previous similar message [ 7291.915133] Lustre: Skipped 1 previous similar message [ 7296.200551] Lustre: server umount lustre-MDT0000 complete [ 7297.007422] LustreError: 254872:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7297.039923] LustreError: 254872:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 7300.508379] LustreError: 254854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789454825 with bad export cookie 14827083819233823315 [ 7300.514086] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7300.520565] LustreError: 254854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7300.937581] Lustre: server umount lustre-MDT0001 complete [ 7313.657715] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 7317.238256] Key type lgssc unregistered [ 7317.621049] LNet: 260283:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7317.626896] LNetError: 260283:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7318.697180] LNet: Removed LNI 192.168.203.127@tcp [ 7319.842636] Key type .llcrypt unregistered [ 7319.848167] Key type ._llcrypt unregistered [ 7349.679571] Key type ._llcrypt registered [ 7349.681990] Key type .llcrypt registered [ 7349.762847] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 7350.459152] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7350.486065] alg: No test for adler32 (adler32-zlib) [ 7351.546254] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 7351.789907] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 7353.519465] Key type lgssc registered [ 7354.951259] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7363.817343] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7363.868345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7365.622248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7371.124294] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 7375.867090] LustreError: 261696:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7375.903949] LustreError: 261696:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7380.965579] LustreError: 261697:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7384.570445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7385.257232] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7390.861453] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 7399.630771] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7406.083767] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7412.941157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7413.314661] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7419.458066] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 7425.405512] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7435.817373] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7461.344769] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7461.345444] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7461.352096] LustreError: Skipped 1 previous similar message [ 7461.353050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7461.398015] Lustre: Skipped 1 previous similar message [ 7464.496942] Lustre: server umount lustre-OST0000 complete [ 7471.585947] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7471.610934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7471.620518] Lustre: Skipped 2 previous similar messages [ 7474.826249] Lustre: server umount lustre-MDT0000 complete [ 7476.705536] LustreError: 261696:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7476.725929] LustreError: 261696:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7478.447787] LustreError: 261677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789455003 with bad export cookie 7181797908094391745 [ 7478.451259] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7478.464416] LustreError: 261677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7478.980744] Lustre: server umount lustre-MDT0001 complete [ 7491.446242] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 7494.271576] Key type lgssc unregistered [ 7494.649795] LNet: 266026:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7494.658454] LNetError: 266026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7494.677726] LNet: Removed LNI 192.168.203.127@tcp [ 7495.925325] Key type .llcrypt unregistered [ 7495.929601] Key type ._llcrypt unregistered [ 7514.997418] Key type ._llcrypt registered [ 7515.001792] Key type .llcrypt registered [ 7515.150577] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 7516.263560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7516.407945] alg: No test for adler32 (adler32-zlib) [ 7517.627584] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 7517.920882] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 7519.591221] Key type lgssc registered [ 7521.141462] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7530.440576] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7530.491147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7532.093312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7536.990109] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 7542.251370] LustreError: 267246:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7542.268651] LustreError: 267246:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7546.797738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7547.244589] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7552.923368] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 7563.264457] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7572.345233] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7580.232370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7580.797323] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7588.448590] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 7597.319394] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7602.304631] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7615.900645] Lustre: server umount lustre-OST0000 complete [ 7623.148363] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7623.169367] Lustre: Skipped 1 previous similar message [ 7623.176064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7628.260282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7628.270341] Lustre: Skipped 2 previous similar messages [ 7630.470260] Lustre: server umount lustre-MDT0000 complete [ 7633.384578] LustreError: 267973:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7633.386086] Lustre: 266557:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789455142/real 1789455142] req@000000007394a35b x1876379610151296/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789455158 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7633.419908] LustreError: 267973:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7633.481265] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7634.847913] LustreError: 267226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789455159 with bad export cookie 6622508829508856161 [ 7634.862621] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7634.872724] LustreError: 267226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7635.524333] Lustre: server umount lustre-MDT0001 complete [ 7647.844626] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 7650.807095] Key type lgssc unregistered [ 7651.058635] LNet: 270926:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7651.077687] LNetError: 270926:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7652.136087] LNet: Removed LNI 192.168.203.127@tcp [ 7653.365821] Key type .llcrypt unregistered [ 7653.367609] Key type ._llcrypt unregistered [ 7680.315026] Key type ._llcrypt registered [ 7680.317367] Key type .llcrypt registered [ 7680.474146] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 7682.100941] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7682.292276] alg: No test for adler32 (adler32-zlib) [ 7683.320815] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 7683.637846] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 7685.463217] Key type lgssc registered [ 7687.527574] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7697.937191] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7697.974841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7700.101765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7707.911612] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 7709.160668] LustreError: 272344:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7709.187838] LustreError: 272344:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7714.276687] LustreError: 272345:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7719.393220] LustreError: 272344:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7720.700467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7721.318519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7727.853710] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 7740.923458] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7748.949534] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7756.750679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7757.207224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7763.872624] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 7769.456565] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7771.888516] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7800.293588] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7800.293928] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7800.346648] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7800.364044] Lustre: Skipped 1 previous similar message [ 7803.751793] Lustre: server umount lustre-OST0000 complete [ 7808.480883] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7808.499136] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7808.506193] Lustre: Skipped 1 previous similar message [ 7808.512158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7810.535574] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7810.537444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7810.577828] Lustre: Skipped 1 previous similar message [ 7814.792579] Lustre: server umount lustre-MDT0000 complete [ 7815.665685] LustreError: 272345:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7815.701484] LustreError: 272345:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7819.261656] LustreError: 272324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789455344 with bad export cookie 4999670856628418869 [ 7819.266335] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7819.284684] LustreError: 272324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7819.870202] Lustre: server umount lustre-MDT0001 complete [ 7832.751911] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 7836.172591] Key type lgssc unregistered [ 7836.436641] LNet: 276653:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7836.447241] LNetError: 276653:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7836.461416] LNet: Removed LNI 192.168.203.127@tcp [ 7837.354527] Key type .llcrypt unregistered [ 7837.357486] Key type ._llcrypt unregistered [ 7857.233849] Key type ._llcrypt registered [ 7857.237096] Key type .llcrypt registered [ 7857.428817] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 7858.736060] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7858.757287] alg: No test for adler32 (adler32-zlib) [ 7860.300857] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 7860.772147] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 7862.591932] Key type lgssc registered [ 7863.952768] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7872.866267] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7872.889781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7874.746686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7880.261116] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 7884.273323] LustreError: 277888:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7884.291307] LustreError: 277888:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7889.377736] LustreError: 277890:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7890.089045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7890.513981] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7896.138384] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 7903.511823] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7910.203860] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7917.325724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7917.695782] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7919.758581] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 7925.669807] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 7937.097671] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7955.939280] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7955.942870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7955.960584] Lustre: Skipped 1 previous similar message [ 7958.541641] Lustre: server umount lustre-OST0000 complete [ 7966.199023] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7966.201848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7966.211912] Lustre: Skipped 1 previous similar message [ 7966.239406] Lustre: Skipped 2 previous similar messages [ 7969.567916] Lustre: server umount lustre-MDT0000 complete [ 7971.298643] LustreError: 277887:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7971.320693] LustreError: 277887:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7973.789109] LustreError: 278647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789455498 with bad export cookie 861815044703564290 [ 7973.811968] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7974.261063] Lustre: server umount lustre-MDT0001 complete [ 7987.791850] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 7991.321586] Key type lgssc unregistered [ 7991.678560] LNet: 281588:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7991.685536] LNetError: 281588:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7991.704492] LNet: Removed LNI 192.168.203.127@tcp [ 7992.503103] Key type .llcrypt unregistered [ 7992.506872] Key type ._llcrypt unregistered [ 8008.173219] Key type ._llcrypt registered [ 8008.175929] Key type .llcrypt registered [ 8008.424670] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8026.818934] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [ 8043.008544] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 8044.023884] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8044.229247] alg: No test for adler32 (adler32-zlib) [ 8045.393604] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 8045.677928] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 8047.367284] Key type lgssc registered [ 8048.538802] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8081.356384] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8081.399951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8082.679243] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8082.721225] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8082.814208] Lustre: lustre-MDT0000: new disk, initializing [ 8082.893638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8082.913766] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8087.069733] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 8097.451310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8097.643810] Lustre: 285136:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8097.751991] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8097.759754] Lustre: Skipped 1 previous similar message [ 8097.909344] Lustre: lustre-MDT0001: new disk, initializing [ 8098.016304] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8098.068583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8098.089671] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8103.026265] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 8112.643126] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8119.102376] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8126.728753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8127.041991] Lustre: lustre-OST0000: new disk, initializing [ 8127.045251] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8127.050937] Lustre: 287107:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8127.162447] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8133.670819] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 8136.292054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8136.303737] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8136.442434] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8143.299901] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8147.714275] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8147.973805] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8153.197204] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8153.407515] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8157.165183] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8157.171609] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8157.174371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8157.178019] LustreError: Skipped 1 previous similar message [ 8157.209993] Lustre: Skipped 1 previous similar message [ 8161.555601] Lustre: server umount lustre-OST0000 complete [ 8167.402349] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8167.434380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8167.465094] Lustre: Skipped 2 previous similar messages [ 8172.524942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8172.543745] Lustre: Skipped 1 previous similar message [ 8173.044702] Lustre: server umount lustre-MDT0000 complete [ 8178.325966] LustreError: 285130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789455703 with bad export cookie 635191364164374975 [ 8178.342306] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8178.355353] LustreError: 285130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8178.959069] Lustre: server umount lustre-MDT0001 complete [ 8193.430916] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 03:01:56 (1789455716) [ 8201.985626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8202.485604] LustreError: 289529:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8202.513307] LustreError: 289529:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8202.577413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8207.841985] LustreError: 289530:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8207.848237] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 8212.968877] LustreError: 289529:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8217.994321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8218.084813] LustreError: 289530:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8218.500043] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8223.767852] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 8231.848807] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8237.994451] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8244.906911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8245.217681] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8251.715157] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 8260.975606] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8277.989395] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8277.993092] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8278.027609] Lustre: Skipped 1 previous similar message [ 8278.049106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8279.018803] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8282.382472] Lustre: server umount lustre-OST0000 complete [ 8289.251639] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8289.257573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8289.273849] Lustre: Skipped 2 previous similar messages [ 8289.293328] Lustre: Skipped 3 previous similar messages [ 8293.047202] Lustre: server umount lustre-MDT0000 complete [ 8294.369750] LustreError: 292023:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8294.386595] LustreError: 292023:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8296.694860] LustreError: 289511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789455821 with bad export cookie 635191364164375857 [ 8296.696846] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8296.716496] LustreError: 289511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8297.156710] Lustre: server umount lustre-MDT0001 complete [ 8307.071728] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 8310.095867] Key type lgssc unregistered [ 8310.369736] LNet: 293084:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8310.377647] LNetError: 293084:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8310.410131] LNet: Removed LNI 192.168.203.127@tcp [ 8311.257882] Key type .llcrypt unregistered [ 8311.260124] Key type ._llcrypt unregistered [ 8339.086913] Key type ._llcrypt registered [ 8339.089679] Key type .llcrypt registered [ 8339.191741] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [ 8354.175233] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 8355.347807] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8355.389858] alg: No test for adler32 (adler32-zlib) [ 8356.500790] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 8356.758381] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 8358.455425] Key type lgssc registered [ 8359.859541] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8407.049584] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8407.084161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8408.354463] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8408.395672] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8408.518492] Lustre: lustre-MDT0000: new disk, initializing [ 8408.662776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8408.800195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8415.230587] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 8428.475855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8428.706508] Lustre: 296909:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8428.785671] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8428.800428] Lustre: Skipped 1 previous similar message [ 8428.990722] Lustre: lustre-MDT0001: new disk, initializing [ 8429.198697] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8429.332920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8429.368386] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8435.142786] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 8445.661181] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8452.013299] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8459.508722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8459.894372] Lustre: lustre-OST2710: new disk, initializing [ 8459.900228] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8459.912300] Lustre: 298879:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8460.019147] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8466.816386] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 8467.546363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8467.561536] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8467.692656] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8477.889846] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8485.281038] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8485.427051] Lustre: lustre-OST03e8: new disk, initializing [ 8485.434398] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8485.439668] Lustre: 299944:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8485.527548] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8492.332681] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 8494.825148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8494.841668] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8495.008293] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8500.325207] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8507.878651] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8508.229724] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8515.106867] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8515.363547] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8520.402884] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8520.725573] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8525.498911] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8525.752301] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8531.431361] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8531.439968] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8531.450394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8531.824185] Lustre: server umount lustre-MDT0000 complete [ 8535.765276] LustreError: 296901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789456060 with bad export cookie 1052766246388119259 [ 8535.770973] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8535.780160] LustreError: 296901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8536.033635] Lustre: lustre-MDT0001-lwp-OST2710: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8536.045736] Lustre: Skipped 3 previous similar messages [ 8536.259831] Lustre: server umount lustre-MDT0001 complete [ 8542.978673] Lustre: server umount lustre-OST2710 complete [ 8547.231460] Lustre: server umount lustre-OST03e8 complete [ 8569.382996] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [ 8580.105848] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 8619.312289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8619.634379] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8619.741106] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8619.890468] Lustre: lustre-MDT0000: new disk, initializing [ 8620.027487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8620.062661] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8626.064084] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 8637.525312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8637.764553] Lustre: 305193:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8637.835897] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8637.841744] Lustre: Skipped 1 previous similar message [ 8637.956808] Lustre: lustre-MDT0001: new disk, initializing [ 8638.076552] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8638.118977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8638.135516] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8642.869799] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 8653.033823] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8658.718481] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8666.016813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8666.287898] Lustre: lustre-OST0000: new disk, initializing [ 8666.291402] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8666.296317] Lustre: 307160:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8666.386582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8673.579797] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 8676.467821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8676.483973] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8676.639935] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8683.187331] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8688.204795] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8688.466975] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8693.462906] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8693.682094] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8697.312601] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8697.320428] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8697.342657] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8697.354112] Lustre: Skipped 1 previous similar message [ 8701.877696] Lustre: server umount lustre-OST0000 complete [ 8707.563741] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8707.565454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8707.586859] Lustre: Skipped 2 previous similar messages [ 8707.598971] Lustre: Skipped 1 previous similar message [ 8712.016718] Lustre: server umount lustre-MDT0000 complete [ 8712.676586] LustreError: 305204:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8712.705410] LustreError: 305204:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 8715.691987] LustreError: 305186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789456240 with bad export cookie 1052766246388120939 [ 8715.703828] LustreError: 305186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8715.703893] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8716.027069] Lustre: server umount lustre-MDT0001 complete [ 8725.515594] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 03:10:49 (1789456249) [ 8726.892819] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8728.553438] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 03:10:52 (1789456252) [ 8732.200331] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 8759.153575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8759.423090] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8759.456736] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8759.671485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8764.196286] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 8772.208479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8772.358795] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8772.406899] Lustre: 311020:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8777.254237] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 8798.588845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8798.899138] Lustre: Denying initial registration attempt from nid 192.168.203.127@tcp, specified as failover [ 8798.908484] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8798.921129] LustreError: 312475:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -99 [ 8798.945038] LustreError: 312475:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 8798.963017] LustreError: 312475:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8799.030729] Lustre: server umount lustre-OST0000 complete [ 8799.040127] LustreError: 312475:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8808.418890] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8808.427895] LustreError: Skipped 1 previous similar message [ 8808.433140] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8808.445698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8813.804457] Lustre: server umount lustre-MDT0000 complete [ 8815.592412] LustreError: 311034:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8815.613802] LustreError: 311034:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8818.022918] LustreError: 311014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789456342 with bad export cookie 1052766246388121814 [ 8818.040070] LustreError: 311014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8818.052910] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8818.380215] Lustre: server umount lustre-MDT0001 complete [ 8830.052960] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 8832.765792] Key type lgssc unregistered [ 8833.094264] LNet: 313596:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8833.105183] LNetError: 313596:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8833.136083] LNet: Removed LNI 192.168.203.127@tcp [ 8834.004710] Key type .llcrypt unregistered [ 8834.009145] Key type ._llcrypt unregistered [ 8846.057986] Key type ._llcrypt registered [ 8846.061911] Key type .llcrypt registered [ 8846.191740] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 8846.897136] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8846.931465] alg: No test for adler32 (adler32-zlib) [ 8847.891823] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 8848.074534] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 8849.696771] Key type lgssc registered [ 8850.683812] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8895.672925] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 8903.910351] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8903.939513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8905.215315] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8905.253267] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8905.482359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8909.925419] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 8917.173028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8917.299064] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8917.348077] Lustre: 316356:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8917.614891] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8922.120524] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 8929.521469] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8934.655189] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8954.417413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8954.696050] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8954.866587] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8961.834188] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 8969.464403] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8972.263083] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8972.277242] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8972.304306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8972.777050] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8972.779436] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8977.812724] Lustre: server umount lustre-OST0000 complete [ 8983.010099] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8983.029073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8983.034962] Lustre: Skipped 2 previous similar messages [ 8988.130930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8988.138212] Lustre: Skipped 1 previous similar message [ 8992.232688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8992.243746] Lustre: Skipped 1 previous similar message [ 8996.319172] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8996.526080] Lustre: server umount lustre-MDT0000 complete [ 8999.895673] LustreError: 316348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789456524 with bad export cookie 7701984113645427125 [ 8999.900341] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8999.919840] LustreError: 316348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9000.287833] Lustre: server umount lustre-MDT0001 complete [ 9011.202455] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 9014.327365] Key type lgssc unregistered [ 9014.681593] LNet: 319930:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9014.687990] LNetError: 319930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9014.720792] LNet: Removed LNI 192.168.203.127@tcp [ 9015.968701] Key type .llcrypt unregistered [ 9015.972634] Key type ._llcrypt unregistered [ 9045.016274] Key type ._llcrypt registered [ 9045.018585] Key type .llcrypt registered [ 9045.145807] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 9046.429333] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9046.444903] alg: No test for adler32 (adler32-zlib) [ 9047.509910] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 9047.819052] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 9049.511184] Key type lgssc registered [ 9050.948771] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9058.780565] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9058.803635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9060.330282] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9065.116155] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 9070.567076] LustreError: 321347:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9070.597895] LustreError: 321347:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9074.204908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9074.872548] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9079.896849] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 9087.887750] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9094.320738] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9102.835825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9103.273943] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9109.234555] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 9116.132335] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9130.976473] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9130.987827] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9131.021735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9134.397151] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.27@tcp (stopping) [ 9135.080171] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9135.108725] Lustre: Skipped 1 previous similar message [ 9137.073552] Lustre: server umount lustre-MDT0000 complete [ 9139.484340] LustreError: 321342:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9140.976737] LustreError: 321327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789456665 with bad export cookie 1808175650979893093 [ 9140.982349] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9140.990679] LustreError: 321327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9144.592549] LustreError: 321343:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9144.625952] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.27@tcp (stopping) [ 9144.628248] LustreError: 321343:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9144.657900] Lustre: Skipped 3 previous similar messages [ 9145.314256] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9147.431418] Lustre: server umount lustre-MDT0001 complete [ 9154.175061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9162.933672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9166.754522] LustreError: 320654:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000009effc1f x1876381214482944/t0(0) o250->MGC192.168.203.127@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9167.142174] LustreError: 323180:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9167.158731] LustreError: 323180:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9167.221233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9167.314058] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9172.292099] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 9175.344407] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9175.344748] LustreError: 323181:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9175.380803] LustreError: 323181:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9182.158789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9182.501914] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9185.595715] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9188.074488] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 9196.282570] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9198.743249] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9206.221283] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9209.142774] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9218.530667] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9218.533067] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9218.554309] LustreError: Skipped 1 previous similar message [ 9218.557212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9218.557224] Lustre: Skipped 1 previous similar message [ 9218.624797] Lustre: Skipped 1 previous similar message [ 9226.728230] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9226.737034] Lustre: Skipped 4 previous similar messages [ 9232.351134] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9232.553947] Lustre: server umount lustre-OST0000 complete [ 9236.960269] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9236.963349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9236.977940] Lustre: Skipped 1 previous similar message [ 9236.992941] Lustre: Skipped 3 previous similar messages [ 9242.703643] Lustre: server umount lustre-MDT0000 complete [ 9246.581748] LustreError: 321327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789456771 with bad export cookie 1808175650979900653 [ 9246.585847] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9246.596245] LustreError: 321327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9247.001742] Lustre: server umount lustre-MDT0001 complete [ 9257.662931] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 9260.671467] Key type lgssc unregistered [ 9261.023494] LNet: 327636:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9261.031354] LNetError: 327636:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9262.060278] LNet: Removed LNI 192.168.203.127@tcp [ 9262.897055] Key type .llcrypt unregistered [ 9262.900542] Key type ._llcrypt unregistered [ 9287.149603] Key type ._llcrypt registered [ 9287.153320] Key type .llcrypt registered [ 9287.274408] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 9288.636096] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9288.651086] alg: No test for adler32 (adler32-zlib) [ 9289.718408] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 9289.972277] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 9291.679248] Key type lgssc registered [ 9292.645140] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9300.298371] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9300.319990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9301.744249] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9305.886445] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 9311.218316] LustreError: 329048:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9311.248071] LustreError: 329048:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9315.677728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9316.027208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9320.753096] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 9327.900894] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9334.064205] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9340.388357] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9340.406601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9340.414209] Lustre: Skipped 1 previous similar message [ 9343.724556] Lustre: server umount lustre-MDT0000 complete [ 9345.507743] LustreError: 330063:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9345.542046] LustreError: 330063:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9347.357927] LustreError: 329030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789456872 with bad export cookie 12906662143265514400 [ 9347.363163] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9347.369884] LustreError: 329030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9347.684655] Lustre: server umount lustre-MDT0001 complete [ 9362.638067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9362.982783] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9363.012817] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9363.211115] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9367.317737] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 9374.894203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9375.037599] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9375.089954] Lustre: 331636:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9375.384256] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9380.112324] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 9388.490730] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9393.669769] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9404.904105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9405.282262] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9405.291919] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9405.310841] LustreError: 333522:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 9405.317078] LustreError: 333522:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 9405.322498] LustreError: 333522:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9405.375295] Lustre: server umount lustre-OST0000 complete [ 9405.377832] LustreError: 333522:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9417.007791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9417.161364] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9417.411246] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9424.800968] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 9425.086840] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9434.530062] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9443.632446] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9443.768701] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9443.874000] Lustre: lustre-OST0001: new disk, initializing [ 9443.882059] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9443.890581] Lustre: 335064:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9444.010906] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9452.764439] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 9453.198948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9453.205394] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9453.300183] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9464.625914] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9468.901388] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9468.907623] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9468.908864] LustreError: Skipped 1 previous similar message [ 9468.914827] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9468.926172] Lustre: Skipped 2 previous similar messages [ 9474.035311] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9474.044089] Lustre: Skipped 2 previous similar messages [ 9479.138116] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9479.151957] Lustre: Skipped 1 previous similar message [ 9482.207145] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9482.321865] Lustre: server umount lustre-OST0001 complete [ 9484.269772] LustreError: 333988:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9484.308136] LustreError: 333988:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9486.824473] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9486.838821] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9486.858782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9489.380183] LustreError: 333990:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9489.383592] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9489.418773] LustreError: 333990:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9492.778507] Lustre: server umount lustre-OST0000 complete [ 9498.080436] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9498.097112] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9498.151506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9498.166103] Lustre: Skipped 2 previous similar messages [ 9502.793665] Lustre: server umount lustre-MDT0000 complete [ 9504.740493] LustreError: 331648:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9504.763527] LustreError: 331648:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9507.037212] LustreError: 331629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789457032 with bad export cookie 12906662143265514953 [ 9507.037645] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9507.054880] LustreError: 331629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9507.414418] Lustre: server umount lustre-MDT0001 complete [ 9526.493300] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [ 9530.038321] Key type lgssc unregistered [ 9530.387481] LNet: 337039:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9530.399535] LNetError: 337039:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9530.445211] LNet: Removed LNI 192.168.203.127@tcp [ 9531.537496] Key type .llcrypt unregistered [ 9531.540369] Key type ._llcrypt unregistered [ 9612.174794] Key type ._llcrypt registered [ 9612.178854] Key type .llcrypt registered [ 9612.349801] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [ 9628.202846] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [ 9629.610345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9629.675353] alg: No test for adler32 (adler32-zlib) [ 9630.747553] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [ 9631.021409] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [ 9632.680322] Key type lgssc registered [ 9633.674738] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9666.665338] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9666.682209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9667.984964] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9668.011438] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9668.112149] Lustre: lustre-MDT0000: new disk, initializing [ 9668.190751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9668.213925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9672.596086] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 9684.784082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9684.934115] Lustre: 341972:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9685.008688] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9685.012401] Lustre: Skipped 1 previous similar message [ 9685.106794] Lustre: lustre-MDT0001: new disk, initializing [ 9685.206437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9685.257360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9685.265230] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9691.467903] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 9703.280773] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9710.659871] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9717.874387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9718.261578] Lustre: lustre-OST0000: new disk, initializing [ 9718.265929] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9718.279409] Lustre: 343938:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9718.364571] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9725.694386] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 9727.128595] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9727.139366] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9727.265019] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9736.411671] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9741.323155] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9741.602791] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9747.506188] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9747.799730] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9753.056399] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9753.056467] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9753.059453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9753.073629] LustreError: Skipped 1 previous similar message [ 9753.093052] Lustre: Skipped 1 previous similar message [ 9756.556711] Lustre: server umount lustre-OST0000 complete [ 9762.272447] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9762.277859] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9762.295632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9762.304448] Lustre: Skipped 1 previous similar message [ 9763.303074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9763.304740] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9766.755884] Lustre: server umount lustre-MDT0000 complete [ 9770.705184] LustreError: 341962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789457295 with bad export cookie 3314961856827320343 [ 9770.708688] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9770.715735] LustreError: 341962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9771.131575] Lustre: server umount lustre-MDT0001 complete [ 9781.131457] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 03:28:24 (1789457304) [ 9791.417697] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 03:28:34 (1789457314) [ 9802.257382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9803.070921] LustreError: 346698:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9803.102267] LustreError: 346698:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9803.202570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9808.361331] LustreError: 346699:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9809.754299] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 9813.479717] LustreError: 346698:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9818.596379] LustreError: 346699:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9820.832885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9821.408248] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9828.133119] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 9838.538122] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9845.689873] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9853.089852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9853.525633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9860.649810] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 9868.682983] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9881.869208] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9883.911672] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9885.924605] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9888.068147] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9890.320775] Lustre: Failing over lustre-MDT0000 [ 9890.614449] Lustre: server umount lustre-MDT0000 complete [ 9892.833291] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9892.843026] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9892.863688] LustreError: 346698:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9903.010658] LustreError: 346699:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9903.032744] LustreError: 346699:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 9910.239368] Lustre: 340193:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789457419/real 1789457419] req@000000007e378491 x1876381825837184/t0(0) o400->MGC192.168.203.127@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789457435 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9910.271872] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9910.608721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9920.483785] LustreError: 340190:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b8280067 x1876381825842560/t0(0) o250->MGC192.168.203.127@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9920.542695] LustreError: 346694:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9920.585144] LustreError: 346694:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 15 previous similar messages [ 9921.112733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9921.212736] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9921.813050] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9926.130113] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9926.156043] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 9926.248825] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9927.611221] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 9938.919561] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9940.443859] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9944.447799] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9946.735114] Lustre: Failing over lustre-MDT0000 [ 9946.996050] Lustre: server umount lustre-MDT0000 complete [ 9951.716440] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9951.744062] Lustre: Skipped 4 previous similar messages [ 9956.838888] LustreError: 349141:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9956.852465] LustreError: 349141:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 9967.252138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9967.405373] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9967.657203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9967.759047] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9968.915381] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9972.739796] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9972.744481] Lustre: Skipped 2 previous similar messages [ 9972.797592] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 9972.837969] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [ 9972.854069] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9982.368272] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9983.974953] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9993.184976] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9993.188966] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9993.200871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9993.200990] Lustre: Skipped 1 previous similar message [ 9993.241056] Lustre: Skipped 2 previous similar messages [ 9996.395976] Lustre: server umount lustre-MDT0000 complete [10000.007309] LustreError: 346679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789457524 with bad export cookie 3314961856827323052 [10000.027399] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10000.028705] LustreError: 346679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10000.432333] Lustre: server umount lustre-MDT0001 complete [10014.896657] Lustre: server umount lustre-OST0000 complete [10034.024402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10034.810811] LustreError: 353063:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10034.840163] LustreError: 353063:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 12 previous similar messages [10034.892916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10039.979902] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [10050.354296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10055.771724] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [10064.608645] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10070.335209] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10077.261864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10079.671748] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10084.017701] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [10093.808717] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10105.315090] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10105.318696] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10105.326503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10105.329424] Lustre: Skipped 2 previous similar messages [10111.533575] Lustre: server umount lustre-OST0000 complete [10116.587202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10116.601383] Lustre: Skipped 3 previous similar messages [10130.399375] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10130.708822] Lustre: server umount lustre-MDT0000 complete [10134.539795] LustreError: 353045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789457659 with bad export cookie 3314961856827323801 [10134.549407] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10134.557981] LustreError: 353045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [10134.960615] Lustre: server umount lustre-MDT0001 complete [10145.026676] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [10148.410987] Key type lgssc unregistered [10148.717377] LNet: 356596:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10148.727964] LNetError: 356596:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10148.754537] LNet: Removed LNI 192.168.203.127@tcp [10150.045054] Key type .llcrypt unregistered [10150.047144] Key type ._llcrypt unregistered [10182.829748] Key type ._llcrypt registered [10182.832765] Key type .llcrypt registered [10182.964104] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [10184.416345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10184.432307] alg: No test for adler32 (adler32-zlib) [10185.676686] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [10186.002611] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [10187.775864] Key type lgssc registered [10189.007161] Lustre: Echo OBD driver; http://www.lustre.org/ [10197.526707] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10197.558441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10199.210715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10204.367218] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [10209.775223] LustreError: 358110:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10209.806670] LustreError: 358110:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10213.386538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10213.859392] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10219.695573] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [10228.873818] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10235.964912] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10243.755390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10244.403151] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10245.465258] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10252.439322] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [10259.961948] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10270.180708] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10270.191424] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10270.215809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10271.985974] Lustre: server umount lustre-MDT0000 complete [10274.279043] LustreError: 358111:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10274.283875] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10274.303684] LustreError: 358111:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10274.346286] Lustre: Skipped 1 previous similar message [10276.404272] LustreError: 358092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789457801 with bad export cookie 379115252662303327 [10276.412389] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10276.419246] LustreError: 358092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10276.818361] Lustre: server umount lustre-MDT0001 complete [10291.314703] Lustre: server umount lustre-OST0000 complete [10306.159750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10306.613351] LustreError: 361824:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10306.638256] LustreError: 361824:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10306.710479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10312.174488] LustreError: 361825:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10312.299586] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [10316.259126] LustreError: 361824:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10322.398282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10322.804238] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10328.912498] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [10339.231363] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10346.698761] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10355.816875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10356.049465] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10363.210989] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [10372.472154] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10376.578528] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10392.037313] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10392.040296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10392.044026] Lustre: Skipped 2 previous similar messages [10392.072524] Lustre: Skipped 2 previous similar messages [10397.156757] LustreError: 364368:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10397.178200] LustreError: 364368:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10397.290244] Lustre: server umount lustre-MDT0000 complete [10400.984394] LustreError: 361806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789457925 with bad export cookie 379115252662304440 [10401.000737] LustreError: 361806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10401.002136] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10401.436255] Lustre: server umount lustre-MDT0001 complete [10415.718732] Lustre: server umount lustre-OST0000 complete [10432.485756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10432.968073] LustreError: 365737:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10433.082214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10437.801252] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [10446.873288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10451.981569] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [10459.649683] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10466.183736] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10472.195987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10472.467409] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10472.477137] Lustre: Skipped 1 previous similar message [10477.608061] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10477.728023] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [10484.585375] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10491.236494] Lustre: server umount lustre-OST0000 complete [10498.034570] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10498.042120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10498.056250] Lustre: Skipped 1 previous similar message [10503.140131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10503.149626] Lustre: Skipped 2 previous similar messages [10505.274135] Lustre: server umount lustre-MDT0000 complete [10508.192538] Lustre: 357419:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789458017/real 1789458017] req@0000000053bd2548 x1876382407814912/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789458033 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10508.220989] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10508.231977] LustreError: 365737:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10508.264417] LustreError: 365737:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [10509.279129] Lustre: 357419:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789458018/real 1789458018] req@000000000cc62403 x1876382407815552/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789458034 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10509.510718] LustreError: 365718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789458034 with bad export cookie 379115252662305462 [10509.526661] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10509.533317] LustreError: 365718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10510.129327] Lustre: server umount lustre-MDT0001 complete [10521.813776] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [10525.029818] Key type lgssc unregistered [10525.397647] LNet: 369267:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10525.407292] LNetError: 369267:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10525.425069] LNet: Removed LNI 192.168.203.127@tcp [10526.329776] Key type .llcrypt unregistered [10526.334933] Key type ._llcrypt unregistered [10554.264486] Key type ._llcrypt registered [10554.268593] Key type .llcrypt registered [10554.421510] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [10555.914669] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10555.945625] alg: No test for adler32 (adler32-zlib) [10557.098078] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [10557.459116] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [10559.183200] Key type lgssc registered [10560.822091] Lustre: Echo OBD driver; http://www.lustre.org/ [10569.347784] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10569.362984] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10569.375504] LustreError: 370706:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10569.402499] LustreError: 370706:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10569.415352] LustreError: 370706:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10569.425528] LustreError: 370706:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10569.437538] LustreError: 370706:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10578.265277] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10578.270855] LustreError: 371103:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10578.299874] LustreError: 371103:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10578.313758] LustreError: 371103:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10578.321475] LustreError: 371103:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10578.331229] LustreError: 371103:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10600.315806] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [10602.959446] Key type lgssc unregistered [10603.161342] LNet: 372106:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10603.165868] LNetError: 372106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10603.180990] LNet: Removed LNI 192.168.203.127@tcp [10604.010937] Key type .llcrypt unregistered [10604.013510] Key type ._llcrypt unregistered [10623.396989] Key type ._llcrypt registered [10623.401648] Key type .llcrypt registered [10623.527056] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [10639.074622] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [10640.575363] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10640.609369] alg: No test for adler32 (adler32-zlib) [10641.668785] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [10642.063439] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [10643.799236] Key type lgssc registered [10645.288597] Lustre: Echo OBD driver; http://www.lustre.org/ [10687.918917] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10687.946147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10689.218846] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10689.246820] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10689.325019] Lustre: lustre-MDT0000: new disk, initializing [10689.399698] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10689.436677] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10693.999625] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [10705.706053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10705.806395] Lustre: 375261:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10705.881826] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10705.892586] Lustre: Skipped 1 previous similar message [10705.982888] Lustre: lustre-MDT0001: new disk, initializing [10706.131578] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10706.175862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10706.201855] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10710.850606] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [10721.327407] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10727.467675] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10735.595103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10735.978604] Lustre: lustre-OST0000: new disk, initializing [10735.991422] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10735.999613] Lustre: 377230:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10736.135886] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10742.835984] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [10744.413590] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10744.426132] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10744.524063] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10753.080645] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10758.061188] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10758.306908] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10762.804223] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10762.972762] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10765.281551] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10765.282092] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10765.289935] LustreError: Skipped 1 previous similar message [10765.300302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10765.318431] Lustre: Skipped 1 previous similar message [10770.406626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10770.415275] Lustre: Skipped 1 previous similar message [10771.446629] Lustre: server umount lustre-OST0000 complete [10775.521559] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10775.524446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10775.533543] Lustre: Skipped 2 previous similar messages [10775.545064] Lustre: Skipped 1 previous similar message [10780.645164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10780.662693] Lustre: Skipped 1 previous similar message [10781.636844] Lustre: server umount lustre-MDT0000 complete [10785.472481] LustreError: 375251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789458310 with bad export cookie 15945833373640746024 [10785.474596] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10785.484472] LustreError: 375251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10785.861606] Lustre: server umount lustre-MDT0001 complete [10795.522435] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10797.569057] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 03:45:20 (1789458320) [10804.769636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10805.426368] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10810.855724] LustreError: 379690:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10810.884574] LustreError: 379690:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10811.234921] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [10815.980398] LustreError: 379691:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10821.090832] LustreError: 379690:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10821.517538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10822.126807] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10827.558606] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [10837.201430] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10844.141845] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10852.178419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10852.675523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10859.378106] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [10867.539941] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10874.981850] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10875.307610] Lustre: lustre-OST0001: new disk, initializing [10875.312780] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10875.329856] Lustre: 382529:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10875.461755] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10882.262092] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [10883.723466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10883.736305] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10883.851597] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10891.758676] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10898.744489] Lustre: lustre-OST0001: Not available for connect from 192.168.203.27@tcp (stopping) [10899.424847] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10899.439418] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10899.456284] Lustre: Skipped 1 previous similar message [10901.773952] Lustre: server umount lustre-OST0001 complete [10903.833930] LustreError: 382525:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10909.672536] LustreError: 381511:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10909.708426] LustreError: 381511:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10919.908075] LustreError: 381511:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10919.939055] LustreError: 381511:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10921.442235] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10921.461491] LustreError: Skipped 1 previous similar message [10921.468595] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10921.483286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10921.499974] Lustre: Skipped 2 previous similar messages [10923.490449] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10926.576338] Lustre: server umount lustre-OST0000 complete [10934.760430] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10934.766429] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10934.781155] Lustre: Skipped 1 previous similar message [10936.728943] Lustre: server umount lustre-MDT0000 complete [10940.349645] LustreError: 379670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789458465 with bad export cookie 15945833373640746906 [10940.360082] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10940.363752] LustreError: 379670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10946.979862] Lustre: server umount lustre-MDT0001 complete [10957.398573] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [10959.779985] Key type lgssc unregistered [10960.135990] LNet: 384517:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10960.142864] LNetError: 384517:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10960.158866] LNet: Removed LNI 192.168.203.127@tcp [10961.167208] Key type .llcrypt unregistered [10961.171350] Key type ._llcrypt unregistered [11015.680681] Key type ._llcrypt registered [11015.683564] Key type .llcrypt registered [11015.836774] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [11017.533932] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11017.582715] alg: No test for adler32 (adler32-zlib) [11018.744952] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [11019.001815] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [11020.759422] Key type lgssc registered [11022.221370] Lustre: Echo OBD driver; http://www.lustre.org/ [11030.632302] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11030.661476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11031.917776] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11031.945588] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11032.208362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11038.428235] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [11048.027316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11048.103039] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11048.137398] Lustre: 387314:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11048.419453] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11053.029783] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [11061.459776] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11067.565246] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11071.465063] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11071.470282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11071.481049] Lustre: Skipped 1 previous similar message [11071.499879] Lustre: Skipped 1 previous similar message [11075.962836] Lustre: server umount lustre-MDT0000 complete [11080.234296] LustreError: 388057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789458605 with bad export cookie 16224008039553196033 [11080.244217] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11080.260250] LustreError: 388057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11080.690797] Lustre: server umount lustre-MDT0001 complete [11092.308449] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 03:50:15 (1789458615) [11099.480328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11099.885439] LustreError: 389828:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11099.907757] LustreError: 389828:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11099.945680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11104.885798] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [11105.256515] LustreError: 389829:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11110.371037] LustreError: 389828:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11114.857469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11115.395444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11121.417383] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [11132.048403] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11139.316442] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11147.704498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11148.000447] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11148.291629] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11156.800651] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [11165.304930] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11172.378251] Lustre: Permanently deactivating lustre-OST0000 [11172.400508] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11180.103474] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11181.941913] LustreError: 392816:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11181.950591] LustreError: 392816:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11181.957598] LustreError: 392816:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11185.271378] LustreError: 392866:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11185.287448] LustreError: 392866:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11193.378623] Lustre: server umount lustre-OST0000 complete [11195.830824] LustreError: 393115:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11195.835053] LustreError: 393115:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11197.921923] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11197.933972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11202.528399] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11202.533130] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11202.547992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11208.170360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11208.174137] Lustre: Skipped 3 previous similar messages [11212.256657] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11212.540413] Lustre: server umount lustre-MDT0000 complete [11213.280710] LustreError: 390554:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11213.301726] LustreError: 390554:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11216.230681] LustreError: 389808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789458741 with bad export cookie 16224008039553196614 [11216.232607] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11216.251647] LustreError: 389808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11226.811754] Lustre: server umount lustre-MDT0001 complete [11236.676294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11241.602859] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [11244.730975] LustreError: 394277:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11251.301598] Lustre: 394616:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.127@tcp [11253.986828] Lustre: server umount MGS complete [11265.882895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11266.403237] LustreError: 395341:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11266.418746] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11266.422378] Lustre: Skipped 1 previous similar message [11266.492258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11271.042838] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [11276.792777] LustreError: 395342:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11276.817836] LustreError: 395342:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11279.542460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11279.823385] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11284.673637] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [11292.961468] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11298.663769] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11300.452899] Lustre: Permanently reactivating lustre-OST0000 [11300.464942] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11300.469886] Lustre: Skipped 2 previous similar messages [11328.992092] LustreError: 397122:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11336.165118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11336.494675] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11336.507522] Lustre: Skipped 1 previous similar message [11343.264767] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [11351.180808] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11359.201375] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11359.205647] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11359.232895] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11360.226929] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11362.482579] Lustre: server umount lustre-OST0000 complete [11366.885458] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11366.904077] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11366.922031] Lustre: Skipped 1 previous similar message [11370.465689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11370.475403] Lustre: Skipped 4 previous similar messages [11372.964919] Lustre: server umount lustre-MDT0000 complete [11375.585185] LustreError: 395341:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11375.604490] LustreError: 395341:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11377.139733] LustreError: 395321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789458902 with bad export cookie 16224008039553197664 [11377.148421] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11377.151254] LustreError: 395321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11377.671651] Lustre: server umount lustre-MDT0001 complete [11390.701680] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [11394.144237] Key type lgssc unregistered [11394.597740] LNet: 399707:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11394.607100] LNetError: 399707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11394.652939] LNet: Removed LNI 192.168.203.127@tcp [11395.769947] Key type .llcrypt unregistered [11395.775350] Key type ._llcrypt unregistered [11415.763659] Key type ._llcrypt registered [11415.765414] Key type .llcrypt registered [11415.902712] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [11432.509724] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [11433.765840] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11433.801151] alg: No test for adler32 (adler32-zlib) [11434.863605] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [11435.131582] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [11436.808122] Key type lgssc registered [11438.062978] Lustre: Echo OBD driver; http://www.lustre.org/ [11474.974050] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11474.993606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11476.157891] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11476.179656] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11476.260678] Lustre: lustre-MDT0000: new disk, initializing [11476.326312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11476.384037] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11481.257993] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [11494.976779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11495.136577] Lustre: 402862:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11495.206942] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11495.220551] Lustre: Skipped 1 previous similar message [11495.303356] Lustre: lustre-MDT0001: new disk, initializing [11495.383308] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11495.410901] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11495.424376] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11499.967719] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [11509.351567] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11515.514472] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11521.768042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11522.302431] Lustre: lustre-OST0000: new disk, initializing [11522.309656] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11522.321567] Lustre: 404833:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11522.492840] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11528.610408] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [11531.311334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11531.331801] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11531.422826] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11536.616521] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11540.908384] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11541.109253] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11545.150913] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11545.358807] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11552.225572] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11552.228648] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11552.232122] LustreError: Skipped 1 previous similar message [11552.234525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11552.278605] Lustre: Skipped 1 previous similar message [11553.494742] Lustre: server umount lustre-OST0000 complete [11556.835985] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11556.852098] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11556.901537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11556.921265] Lustre: Skipped 1 previous similar message [11562.464815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11562.471854] Lustre: Skipped 3 previous similar messages [11562.851085] Lustre: server umount lustre-MDT0000 complete [11566.463897] LustreError: 402854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789459091 with bad export cookie 4704858517187491009 [11566.474787] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11566.475178] LustreError: 402854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11566.779293] Lustre: server umount lustre-MDT0001 complete [11575.571486] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11577.402463] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 03:58:20 (1789459100) [11583.995593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11584.676694] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11589.311629] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [11590.124242] LustreError: 407294:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11590.157124] LustreError: 407294:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11594.213221] LustreError: 407293:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11598.036771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11598.509835] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11603.216724] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [11610.774057] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11616.275890] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11623.090112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11623.375254] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11629.825873] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [11636.421055] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11638.083451] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11641.880066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11648.993882] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11649.003788] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11649.031409] Lustre: Skipped 1 previous similar message [11649.041147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11654.070604] Lustre: server umount lustre-OST0000 complete [11660.265764] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11660.280560] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11660.283065] Lustre: Skipped 1 previous similar message [11660.291176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11660.295576] LustreError: Skipped 1 previous similar message [11660.325156] Lustre: Skipped 3 previous similar messages [11664.783805] Lustre: server umount lustre-MDT0000 complete [11665.380587] LustreError: 407293:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11665.411205] LustreError: 407293:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11669.265223] LustreError: 407275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789459194 with bad export cookie 4704858517187491891 [11669.277841] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11669.285419] LustreError: 407275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11669.803480] Lustre: server umount lustre-MDT0001 complete [11683.418293] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [11687.089023] Key type lgssc unregistered [11687.549658] LNet: 411019:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11687.559108] LNetError: 411019:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11687.620914] LNet: Removed LNI 192.168.203.127@tcp [11688.758605] Key type .llcrypt unregistered [11688.763784] Key type ._llcrypt unregistered [11716.444572] Key type ._llcrypt registered [11716.448511] Key type .llcrypt registered [11716.535779] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [11716.802584] Key type .llcrypt unregistered [11716.805384] Key type ._llcrypt unregistered [11736.003109] Key type ._llcrypt registered [11736.005334] Key type .llcrypt registered [11736.161764] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [11737.663922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11737.803307] alg: No test for adler32 (adler32-zlib) [11738.897777] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [11739.155658] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [11740.889911] Key type lgssc registered [11742.137133] Lustre: Echo OBD driver; http://www.lustre.org/ [11749.799822] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11749.851452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11751.278951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11756.517658] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [11761.636664] LustreError: 413236:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11761.648718] LustreError: 413236:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11766.100414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11766.591756] LustreError: 413235:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11766.636330] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11771.873622] LustreError: 413236:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11771.887777] LustreError: 413236:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11773.053131] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [11776.996927] LustreError: 413232:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11777.017083] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11777.020126] LustreError: 413232:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11781.321572] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11782.118292] LustreError: 413235:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11782.135311] LustreError: 413235:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11788.206482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11788.697665] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11794.289743] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [11807.716392] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11807.724124] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11807.750332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11809.250997] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11809.252951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11809.270284] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11809.308343] Lustre: Skipped 1 previous similar message [11811.961746] Lustre: server umount lustre-OST0000 complete [11819.491670] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11819.493159] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11819.517717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11819.527478] Lustre: Skipped 1 previous similar message [11822.153339] Lustre: server umount lustre-MDT0000 complete [11824.616062] LustreError: 413235:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11824.642624] LustreError: 413235:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11825.955667] LustreError: 413215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789459350 with bad export cookie 12169181236002045267 [11825.966256] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11825.966976] LustreError: 413215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11826.302571] Lustre: server umount lustre-MDT0001 complete [11837.375450] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [11840.784567] Key type lgssc unregistered [11841.135186] LNet: 416382:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11841.146154] LNetError: 416382:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11841.169884] LNet: Removed LNI 192.168.203.127@tcp [11842.149557] Key type .llcrypt unregistered [11842.152982] Key type ._llcrypt unregistered [11870.180337] Key type ._llcrypt registered [11870.184332] Key type .llcrypt registered [11870.287894] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [11871.529622] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11871.697883] alg: No test for adler32 (adler32-zlib) [11872.860527] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [11873.097543] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [11874.751197] Key type lgssc registered [11876.389910] Lustre: Echo OBD driver; http://www.lustre.org/ [11884.657431] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11884.689238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11900.319991] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11905.439344] LustreError: 417770:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.127@tcp: failed processing log, type 4: rc = -110 [11932.127280] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11939.640359] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [11948.213750] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11956.522907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11957.254221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11957.359193] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11962.353691] LustreError: 417795:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11962.388652] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [11962.389246] LustreError: 417795:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11966.449798] LustreError: 417796:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11966.479974] LustreError: 417796:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11971.556477] LustreError: 418807:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11971.575248] LustreError: 418807:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11971.681989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11972.181029] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11976.918740] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [11984.631386] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11989.975514] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11997.668302] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11997.671431] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11997.711036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11998.789823] Lustre: server umount lustre-OST0000 complete [12002.786098] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12002.791778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12002.801146] Lustre: Skipped 2 previous similar messages [12002.815663] Lustre: Skipped 2 previous similar messages [12004.685450] Lustre: server umount lustre-MDT0000 complete [12007.905854] LustreError: 418809:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12007.929817] LustreError: 418809:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12008.255771] LustreError: 417776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789459533 with bad export cookie 5970572459942622398 [12008.265781] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12008.268547] LustreError: 417776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12008.597745] Lustre: server umount lustre-MDT0001 complete [12017.911144] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [12020.604914] Key type lgssc unregistered [12020.905243] LNet: 421337:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12020.913444] LNetError: 421337:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12020.933534] LNet: Removed LNI 192.168.203.127@tcp [12021.765980] Key type .llcrypt unregistered [12021.769309] Key type ._llcrypt unregistered [12044.992532] Key type ._llcrypt registered [12044.993985] Key type .llcrypt registered [12045.085797] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [12046.020838] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12046.115691] alg: No test for adler32 (adler32-zlib) [12047.127747] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [12047.494450] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [12049.191696] Key type lgssc registered [12050.643963] Lustre: Echo OBD driver; http://www.lustre.org/ [12058.531701] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12058.548210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12059.882730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12064.274345] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [12069.360469] LustreError: 422750:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12069.388260] LustreError: 422750:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12073.205235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12073.764604] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12078.205524] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [12086.193807] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12091.768502] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12100.637579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12101.434695] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12104.502425] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12107.692045] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [12115.466576] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12123.427411] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.27@tcp (stopping) [12124.131282] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12124.132377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12124.140605] Lustre: Skipped 1 previous similar message [12124.153968] Lustre: Skipped 2 previous similar messages [12126.540552] Lustre: server umount lustre-MDT0000 complete [12128.539579] LustreError: 422747:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12128.964715] LustreError: 423485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.27@tcp arrived at 1789459653 with bad export cookie 15026463146300749051 [12128.983098] LustreError: 423485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12130.866965] LustreError: 422730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789459655 with bad export cookie 15026463146300748162 [12130.875983] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12130.878892] LustreError: 422730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12134.371370] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12134.376393] LustreError: 422745:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12134.377585] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12134.377591] Lustre: Skipped 1 previous similar message [12134.379396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12134.436900] LustreError: 422745:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [12137.184435] Lustre: server umount lustre-OST0000 complete [12140.514727] LustreError: 422750:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12140.531850] LustreError: 422750:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12143.717125] Lustre: server umount lustre-MDT0001 complete [12153.600124] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [12156.268141] Key type lgssc unregistered [12156.548740] LNet: 426434:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12156.554837] LNetError: 426434:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12157.607581] LNet: Removed LNI 192.168.203.127@tcp [12158.495782] Key type .llcrypt unregistered [12158.497821] Key type ._llcrypt unregistered [12182.452141] Key type ._llcrypt registered [12182.454964] Key type .llcrypt registered [12182.574245] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [12183.734911] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12183.783406] alg: No test for adler32 (adler32-zlib) [12184.847627] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [12185.113095] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [12186.775257] Key type lgssc registered [12187.809107] Lustre: Echo OBD driver; http://www.lustre.org/ [12195.522304] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12195.576442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12197.136795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12201.638267] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [12207.588330] LustreError: 427849:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12207.602782] LustreError: 427849:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12209.597105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12209.861497] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12213.637628] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [12220.447304] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12225.521933] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12232.213174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12232.609083] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12238.308991] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [12242.882212] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12245.556463] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12251.113308] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12251.127070] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12251.147455] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12253.157464] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12253.161487] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12256.358981] Lustre: server umount lustre-MDT0001 complete [12258.283891] LustreError: 427849:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12258.298440] LustreError: 427849:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12263.397109] LustreError: 427850:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12263.418914] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12263.444147] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12263.456332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12263.462319] Lustre: Skipped 2 previous similar messages [12268.514191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12268.519126] LustreError: 427849:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12269.161676] Lustre: server umount lustre-OST0000 complete [12273.495134] Lustre: server umount lustre-MDT0000 complete [12289.312402] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [12292.485636] Key type lgssc unregistered [12292.831130] LNet: 431533:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12292.839947] LNetError: 431533:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12292.864873] LNet: Removed LNI 192.168.203.127@tcp [12293.770483] Key type .llcrypt unregistered [12293.772410] Key type ._llcrypt unregistered [12321.333275] Key type ._llcrypt registered [12321.340287] Key type .llcrypt registered [12321.461400] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [12337.007723] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [12337.923550] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12338.026808] alg: No test for adler32 (adler32-zlib) [12339.117542] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [12339.352201] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [12341.007323] Key type lgssc registered [12342.164639] Lustre: Echo OBD driver; http://www.lustre.org/ [12376.332968] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [12392.819677] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12392.854557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12394.124237] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12394.166471] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12394.275988] Lustre: lustre-MDT0000: new disk, initializing [12394.400619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12394.421233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12399.704696] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [12417.478920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12417.839890] Lustre: lustre-OST0000: new disk, initializing [12417.843272] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12417.845573] Lustre: Skipped 1 previous similar message [12417.850165] Lustre: 436562:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12417.920220] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12424.535804] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [12426.360052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12426.378047] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12426.411292] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12433.913695] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12450.387815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12450.499836] Lustre: 435544:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12450.591575] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12450.671584] Lustre: lustre-MDT0001: new disk, initializing [12450.806858] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12450.868561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12450.889498] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12455.718589] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [12465.363529] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12465.526858] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12470.128317] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12470.321927] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12477.416744] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12477.417964] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12477.437030] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12477.449243] Lustre: Skipped 1 previous similar message [12481.606244] Lustre: server umount lustre-MDT0001 complete [12487.657498] LustreError: 438416:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12487.674741] LustreError: 438416:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [12492.772647] LustreError: 435556:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12492.791230] LustreError: 435556:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12497.900557] LustreError: 435551:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12497.928684] LustreError: 435551:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12503.009583] LustreError: 435556:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12503.010673] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12503.011232] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12503.011764] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12503.011767] Lustre: Skipped 2 previous similar messages [12505.697933] Lustre: server umount lustre-OST0000 complete [12508.147975] LustreError: 435557:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12516.488866] Lustre: server umount lustre-MDT0000 complete [12533.836412] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [12537.897547] Key type lgssc unregistered [12538.503761] LNet: 440309:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12538.513091] LNetError: 440309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12538.546839] LNet: Removed LNI 192.168.203.127@tcp [12539.835975] Key type .llcrypt unregistered [12539.840852] Key type ._llcrypt unregistered [12582.143774] Key type ._llcrypt registered [12582.150147] Key type .llcrypt registered [12582.316373] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [12582.627651] Key type .llcrypt unregistered [12582.632333] Key type ._llcrypt unregistered [12599.415850] Key type ._llcrypt registered [12599.418151] Key type .llcrypt registered [12599.519810] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [12600.559543] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12600.731846] alg: No test for adler32 (adler32-zlib) [12601.888339] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [12602.216600] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [12603.991315] Key type lgssc registered [12605.069095] Lustre: Echo OBD driver; http://www.lustre.org/ [12634.504466] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12634.546253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12635.753540] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12635.783047] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12635.845763] Lustre: lustre-MDT0000: new disk, initializing [12635.908479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12635.919801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12640.578833] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [12651.236276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12651.361176] Lustre: 443681:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12651.416129] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12651.421354] Lustre: Skipped 1 previous similar message [12651.473598] Lustre: lustre-MDT0001: new disk, initializing [12651.563927] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12651.608738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12651.625874] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12655.747578] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [12666.714359] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12675.109653] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12683.399069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12683.597180] Lustre: lustre-OST0000: new disk, initializing [12683.607261] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12683.626715] Lustre: 445651:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12683.766807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12691.527309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12691.542888] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12691.726947] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12691.986048] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [12703.079748] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12712.422067] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12712.433309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12712.449627] Lustre: Skipped 2 previous similar messages [12712.455803] Lustre: Skipped 1 previous similar message [12715.610805] Lustre: server umount lustre-MDT0000 complete [12718.960866] LustreError: 444476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789460243 with bad export cookie 17774626999081143189 [12718.961868] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12718.972309] LustreError: 444476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12719.129834] Lustre: server umount lustre-MDT0001 complete [12732.999678] Lustre: server umount lustre-OST0000 complete [12743.344873] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 04:17:46 (1789460266) [12750.954334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12751.488782] LustreError: 447679:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12751.510207] LustreError: 447679:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [12751.547543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12756.288925] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [12756.961207] LustreError: 447680:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12761.061236] LustreError: 447679:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12764.496318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12764.793549] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12768.978867] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [12776.156584] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12781.967863] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12802.093872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12802.342340] Lustre: Found index 0 for lustre-OST0000, updating log [12802.570414] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12809.289744] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [12816.678349] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12822.074208] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12827.112103] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12827.116688] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12827.121503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12827.121511] Lustre: Skipped 1 previous similar message [12827.134578] Lustre: Skipped 1 previous similar message [12828.204901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12828.211089] Lustre: Skipped 2 previous similar messages [12828.787873] Lustre: server umount lustre-OST0000 complete [12846.562687] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12846.571782] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12846.592148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12849.635631] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12850.983107] Lustre: server umount lustre-MDT0000 complete [12854.757226] LustreError: 447674:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12854.786261] LustreError: 447674:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12855.296656] LustreError: 447660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789460380 with bad export cookie 17774626999081144582 [12855.303319] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12855.306252] LustreError: 447660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12855.899781] Lustre: server umount lustre-MDT0001 complete [12864.817681] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 04:19:48 (1789460388) [12865.439110] LNet: 1 local NIs in recovery (showing 1): 192.168.203.127@tcp [13488.169964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13488.720964] LustreError: 454089:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13488.864963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13494.043674] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [13494.243394] LustreError: 454090:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13498.343257] LustreError: 454089:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13503.549123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13504.064090] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13509.184736] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [13518.024777] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13524.144794] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13543.391142] LNet: 1 local NIs in recovery (showing 1): 192.168.203.127@tcp [13544.566656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13545.021106] Lustre: Found index 0 for lustre-OST0000, updating log [13545.527935] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13553.803732] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [13561.298485] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13563.643332] Lustre: server umount lustre-OST0000 complete [13568.673691] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13568.685313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13568.695689] Lustre: Skipped 1 previous similar message [13568.711633] Lustre: Skipped 3 previous similar messages [13569.763146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13569.778146] Lustre: Skipped 1 previous similar message [13570.787205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13570.807047] Lustre: Skipped 1 previous similar message [13572.838091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13572.840653] Lustre: Skipped 2 previous similar messages [13573.858695] LustreError: 454089:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13573.877606] LustreError: 454089:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13573.952488] Lustre: server umount lustre-MDT0000 complete [13577.559871] LustreError: 454070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789461102 with bad export cookie 17774626999081145667 [13577.560718] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13577.577182] LustreError: 454070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13578.016806] Lustre: server umount lustre-MDT0001 complete [13603.871272] LNet: 1 local NIs in recovery (showing 1): 192.168.203.127@tcp [13607.452424] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [13617.210956] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [13652.212216] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 04:32:55 (1789461175) [13670.354710] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [13673.288838] Key type lgssc unregistered [13673.603722] LNet: 460700:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13673.616151] LNetError: 460700:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13804.709836] LNet: Removed LNI 192.168.203.127@tcp [13805.672439] Key type .llcrypt unregistered [13805.675030] Key type ._llcrypt unregistered [13822.766743] Key type ._llcrypt registered [13822.769781] Key type .llcrypt registered [13822.903289] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [13823.831458] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13824.007055] alg: No test for adler32 (adler32-zlib) [13825.117893] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [13825.435443] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [13827.167710] Key type lgssc registered [13828.574939] Lustre: Echo OBD driver; http://www.lustre.org/ [13833.449734] LNet: Added LNI 192.168.203.127@tcp100 [8/256/0/180] [13834.742813] LNet: Added LNI 192.168.203.127@tcp101 [8/256/0/180] [13835.738925] LNet: Added LNI 192.168.203.127@tcp102 [8/256/0/180] [13836.712458] LNet: Added LNI 192.168.203.127@tcp103 [8/256/0/180] [13837.775224] LNet: Added LNI 192.168.203.127@tcp104 [8/256/0/180] [13839.002235] LNet: Added LNI 192.168.203.127@tcp105 [8/256/0/180] [13839.998433] LNet: Added LNI 192.168.203.127@tcp106 [8/256/0/180] [13841.392628] LNet: Added LNI 192.168.203.127@tcp107 [8/256/0/180] [13842.591888] LNet: Added LNI 192.168.203.127@tcp108 [8/256/0/180] [13844.026932] LNet: Added LNI 192.168.203.127@tcp109 [8/256/0/180] [13845.297097] LNet: Added LNI 192.168.203.127@tcp110 [8/256/0/180] [13846.424416] LNet: Added LNI 192.168.203.127@tcp111 [8/256/0/180] [13847.121325] LNet: Added LNI 192.168.203.127@tcp112 [8/256/0/180] [13847.844353] LNet: Added LNI 192.168.203.127@tcp113 [8/256/0/180] [13848.813323] LNet: Added LNI 192.168.203.127@tcp114 [8/256/0/180] [13849.678354] LNet: Added LNI 192.168.203.127@tcp115 [8/256/0/180] [13850.388248] LNet: Added LNI 192.168.203.127@tcp116 [8/256/0/180] [13851.024904] LNet: Added LNI 192.168.203.127@tcp117 [8/256/0/180] [13851.658072] LNet: Added LNI 192.168.203.127@tcp118 [8/256/0/180] [13852.446221] LNet: Added LNI 192.168.203.127@tcp119 [8/256/0/180] [13853.396537] LNet: Added LNI 192.168.203.127@tcp120 [8/256/0/180] [13854.340428] LNet: Added LNI 192.168.203.127@tcp121 [8/256/0/180] [13855.186854] LNet: Added LNI 192.168.203.127@tcp122 [8/256/0/180] [13855.986473] LNet: Added LNI 192.168.203.127@tcp123 [8/256/0/180] [13856.798508] LNet: Added LNI 192.168.203.127@tcp124 [8/256/0/180] [13857.697421] LNet: Added LNI 192.168.203.127@tcp125 [8/256/0/180] [13858.597951] LNet: Added LNI 192.168.203.127@tcp126 [8/256/0/180] [13859.482990] LNet: Added LNI 192.168.203.127@tcp127 [8/256/0/180] [13860.332427] LNet: Added LNI 192.168.203.127@tcp128 [8/256/0/180] [13861.362880] LNet: Added LNI 192.168.203.127@tcp129 [8/256/0/180] [13862.241482] LNet: Added LNI 192.168.203.127@tcp130 [8/256/0/180] [13863.228916] LNet: Added LNI 192.168.203.127@tcp131 [8/256/0/180] [13864.271956] LNet: Added LNI 192.168.203.127@tcp132 [8/256/0/180] [13865.306195] LNet: Added LNI 192.168.203.127@tcp133 [8/256/0/180] [13866.077195] LNet: Added LNI 192.168.203.127@tcp134 [8/256/0/180] [13866.981379] LNet: Added LNI 192.168.203.127@tcp135 [8/256/0/180] [13873.908248] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13873.942631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13875.126417] Lustre: 463731:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.127@tcp: can't add NID 192.168.203.127@tcp130: rc = -75 [13875.265566] LNetError: 463753:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.127 [13875.279542] LNetError: Protocol error connecting to 192.168.203.127@tcp119 at host 192.168.203.127:988: is it running a compatible version of Lustre? [13875.289679] LNetError: Skipped 3 previous similar messages [13875.349226] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13875.434086] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13875.583270] Lustre: lustre-MDT0000: new disk, initializing [13875.618036] LNet: 2 local NIs in recovery (showing 2): 192.168.203.127@tcp128, 192.168.203.127@tcp129 [13875.718856] Lustre: 463788:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.127@tcp: can't add NID 192.168.203.127@tcp130: rc = -75 [13875.733720] Lustre: 463788:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [13875.774737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13875.802820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13876.320639] LNetError: 463763:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.127 [13876.337476] LNetError: 463763:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 7 previous similar messages [13876.348200] LNetError: Protocol error connecting to 192.168.203.127@tcp105 at host 192.168.203.127:988: is it running a compatible version of Lustre? [13876.359580] LNetError: Skipped 4 previous similar messages [13878.689063] LNetError: 461188:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.127 [13878.711262] LNetError: 461188:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 3 previous similar messages [13878.722200] LNetError: Protocol error connecting to 192.168.203.127@tcp128 at host 192.168.203.127:988: is it running a compatible version of Lustre? [13878.732163] LNetError: Skipped 2 previous similar messages [13880.921624] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [13882.784893] LNetError: 463753:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.127 [13882.793084] LNetError: Protocol error connecting to 192.168.203.127@tcp128 at host 192.168.203.127:988: is it running a compatible version of Lustre? [13890.986175] LNetError: 463755:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.127 [13891.003154] LNetError: Protocol error connecting to 192.168.203.127@tcp128 at host 192.168.203.127:988: is it running a compatible version of Lustre? [13891.087212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13891.295586] Lustre: 463748:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13891.365993] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13891.371955] Lustre: Skipped 1 previous similar message [13891.490482] Lustre: lustre-MDT0001: new disk, initializing [13891.528168] Lustre: 464550:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.127@tcp: can't add NID 192.168.203.127@tcp130: rc = -75 [13891.543119] Lustre: 464550:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [13891.674424] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13891.709521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13891.724054] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13896.537776] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [13901.359345] Lustre: 464979:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.127@tcp: can't add NID 192.168.203.127@tcp130: rc = -75 [13901.372847] Lustre: 464979:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13905.314838] LNetError: 461187:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.127 [13905.340057] LNetError: Protocol error connecting to 192.168.203.127@tcp120 at host 192.168.203.127:988: is it running a compatible version of Lustre? [13906.364467] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13912.195400] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13918.847713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13919.329605] Lustre: lustre-OST0000: new disk, initializing [13919.356405] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13919.367160] Lustre: 465724:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13919.529160] Lustre: 465729:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.127@tcp: can't add NID 192.168.203.127@tcp130: rc = -75 [13919.538509] Lustre: 465729:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [13919.595829] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13921.704175] LNetError: 463753:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.127 [13921.727872] LNetError: 463753:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 8 previous similar messages [13921.742234] LNetError: Protocol error connecting to 192.168.203.127@tcp105 at host 192.168.203.127:988: is it running a compatible version of Lustre? [13921.753831] LNetError: Skipped 8 previous similar messages [13926.684267] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [13928.454336] Lustre: 466166:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.127@tcp: can't add NID 192.168.203.127@tcp130: rc = -75 [13928.462988] Lustre: 466166:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13928.523620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13928.549146] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13928.683719] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13936.031372] LNet: 1 local NIs in recovery (showing 1): 192.168.203.127@tcp128 [13936.995711] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13949.420683] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13949.422436] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13949.439043] LustreError: Skipped 1 previous similar message [13949.455120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13951.875489] Lustre: server umount lustre-OST0000 complete [13955.490596] LNetError: 461188:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.127 [13955.506550] LNetError: 461188:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 20 previous similar messages [13955.516986] LNetError: Protocol error connecting to 192.168.203.127@tcp125 at host 192.168.203.127:988: is it running a compatible version of Lustre? [13955.531123] LNetError: Skipped 20 previous similar messages [13958.112453] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13958.125152] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13958.132702] Lustre: Skipped 1 previous similar message [13958.146836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13958.149925] Lustre: Skipped 1 previous similar message [13959.649141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13959.652123] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13962.035092] Lustre: server umount lustre-MDT0000 complete [13965.978335] LustreError: 463741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789461490 with bad export cookie 12888873799446730640 [13965.985714] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13965.992427] LustreError: 463741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13966.389651] Lustre: server umount lustre-MDT0001 complete [13977.962182] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [13981.136322] Key type lgssc unregistered [13981.558309] LNet: 467635:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13981.565222] LNetError: 467635:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13981.572991] LNet: Removed LNI 192.168.203.127@tcp [13982.623152] LNet: Removed LNI 192.168.203.127@tcp100 [13982.626703] LNetError: 467635:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13982.632334] LNetError: 467635:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13982.650410] LNet: Removed LNI 192.168.203.127@tcp101 [13983.711284] LNet: Removed LNI 192.168.203.127@tcp102 [13983.716358] LNetError: 467635:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13983.727597] LNetError: 467635:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13983.742057] LNet: Removed LNI 192.168.203.127@tcp103 [13984.799673] LNet: Removed LNI 192.168.203.127@tcp104 [13985.823557] LNet: Removed LNI 192.168.203.127@tcp105 [13985.829411] LNetError: 467635:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13985.840595] LNetError: 467635:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [13985.849399] LNet: Removed LNI 192.168.203.127@tcp106 [13986.911371] LNet: Removed LNI 192.168.203.127@tcp107 [13987.935285] LNet: Removed LNI 192.168.203.127@tcp108 [13988.959929] LNet: Removed LNI 192.168.203.127@tcp109 [13989.984265] LNet: Removed LNI 192.168.203.127@tcp110 [13989.991427] LNetError: 467635:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13990.001183] LNetError: 467635:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [13990.012463] LNet: Removed LNI 192.168.203.127@tcp111 [13991.071634] LNet: Removed LNI 192.168.203.127@tcp112 [13992.095144] LNet: Removed LNI 192.168.203.127@tcp113 [13993.119722] LNet: Removed LNI 192.168.203.127@tcp114 [13994.143434] LNet: Removed LNI 192.168.203.127@tcp115 [13995.167284] LNet: Removed LNI 192.168.203.127@tcp116 [13996.191147] LNet: Removed LNI 192.168.203.127@tcp117 [13997.216620] LNet: Removed LNI 192.168.203.127@tcp118 [13998.239345] LNet: Removed LNI 192.168.203.127@tcp119 [13998.245449] LNetError: 467635:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13998.255813] LNetError: 467635:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [13998.270963] LNet: Removed LNI 192.168.203.127@tcp120 [13999.327271] LNet: Removed LNI 192.168.203.127@tcp121 [14000.351142] LNet: Removed LNI 192.168.203.127@tcp122 [14001.375233] LNet: Removed LNI 192.168.203.127@tcp123 [14002.399151] LNet: Removed LNI 192.168.203.127@tcp124 [14003.423292] LNet: Removed LNI 192.168.203.127@tcp125 [14004.447144] LNet: Removed LNI 192.168.203.127@tcp126 [14005.473158] LNet: Removed LNI 192.168.203.127@tcp127 [14005.475851] LNet: Removed LNI 192.168.203.127@tcp128 [14006.496108] LNet: Removed LNI 192.168.203.127@tcp129 [14006.499289] LNet: Removed LNI 192.168.203.127@tcp130 [14006.507629] LNet: Removed LNI 192.168.203.127@tcp131 [14006.515492] LNet: Removed LNI 192.168.203.127@tcp132 [14006.522368] LNet: Removed LNI 192.168.203.127@tcp133 [14006.531303] LNet: Removed LNI 192.168.203.127@tcp134 [14006.546065] LNet: Removed LNI 192.168.203.127@tcp135 [14007.249795] Key type .llcrypt unregistered [14007.256402] Key type ._llcrypt unregistered [14029.471464] Key type ._llcrypt registered [14029.476190] Key type .llcrypt registered [14029.595619] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [14044.614719] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [14045.837915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14045.921377] alg: No test for adler32 (adler32-zlib) [14047.028804] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [14047.267161] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [14048.968330] Key type lgssc registered [14049.967774] Lustre: Echo OBD driver; http://www.lustre.org/ [14082.717947] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 04:40:05 (1789461605) [14105.937388] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [14109.135911] Key type lgssc unregistered [14109.480812] LNet: 471721:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14109.487804] LNetError: 471721:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14109.506261] LNet: Removed LNI 192.168.203.127@tcp [14110.557320] Key type .llcrypt unregistered [14110.560638] Key type ._llcrypt unregistered [14130.532786] Key type ._llcrypt registered [14130.535740] Key type .llcrypt registered [14130.667476] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [14131.875774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14131.920307] alg: No test for adler32 (adler32-zlib) [14133.029441] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [14133.338159] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [14135.055457] Key type lgssc registered [14136.360279] Lustre: Echo OBD driver; http://www.lustre.org/ [14144.723076] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14144.748423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14145.964039] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14146.020387] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14146.118347] Lustre: lustre-MDT0000: new disk, initializing [14146.217972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14146.236166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14151.011278] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [14161.125553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14161.222969] Lustre: 472928:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14161.290415] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14161.294367] Lustre: Skipped 1 previous similar message [14161.363945] Lustre: lustre-MDT0001: new disk, initializing [14161.490910] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14161.538320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14161.553946] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14165.809387] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [14175.553114] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14182.101343] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14189.051724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14189.268264] Lustre: lustre-OST0000: new disk, initializing [14189.271745] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14189.277063] Lustre: 474900:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14189.364627] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14195.335606] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [14199.351995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14199.362449] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14199.514739] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14204.693381] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14209.241267] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing lnet_if_list [14214.790403] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14214.817199] LNet: Added LNI 192.168.203.127@tcp10 [8/256/0/180] [14220.423940] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14220.467226] LNet: Added LNI 192.168.203.127@tcp11 [8/256/0/180] [14224.794823] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14224.815894] LNet: Added LNI 192.168.203.127@tcp12 [8/256/0/180] [14229.026797] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14229.054919] LNet: Added LNI 192.168.203.127@tcp13 [8/256/0/180] [14233.618702] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14233.640737] LNet: Added LNI 192.168.203.127@tcp14 [8/256/0/180] [14238.197617] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14238.228656] LNet: Added LNI 192.168.203.127@tcp15 [8/256/0/180] [14243.435173] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14243.481746] LNet: Added LNI 192.168.203.127@tcp16 [8/256/0/180] [14249.163785] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14249.188442] LNet: Added LNI 192.168.203.127@tcp17 [8/256/0/180] [14254.239352] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14254.260512] LNet: Added LNI 192.168.203.127@tcp18 [8/256/0/180] [14260.200707] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14260.238626] LNet: Added LNI 192.168.203.127@tcp19 [8/256/0/180] [14265.669506] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14265.696244] LNet: Added LNI 192.168.203.127@tcp20 [8/256/0/180] [14271.227532] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14271.257793] LNet: Added LNI 192.168.203.127@tcp21 [8/256/0/180] [14276.674992] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14276.727964] LNet: Added LNI 192.168.203.127@tcp22 [8/256/0/180] [14282.160300] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14282.188493] LNet: Added LNI 192.168.203.127@tcp23 [8/256/0/180] [14287.681779] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14287.739907] LNet: Added LNI 192.168.203.127@tcp24 [8/256/0/180] [14293.588985] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14293.617477] LNet: Added LNI 192.168.203.127@tcp25 [8/256/0/180] [14299.995096] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14300.041085] LNet: Added LNI 192.168.203.127@tcp26 [8/256/0/180] [14305.833433] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14305.858065] LNet: Added LNI 192.168.203.127@tcp27 [8/256/0/180] [14311.542524] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14311.605750] LNet: Added LNI 192.168.203.127@tcp28 [8/256/0/180] [14317.311082] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14317.345189] LNet: Added LNI 192.168.203.127@tcp29 [8/256/0/180] [14322.996575] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14323.025154] LNet: Added LNI 192.168.203.127@tcp30 [8/256/0/180] [14329.307230] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14329.334343] LNet: Added LNI 192.168.203.127@tcp31 [8/256/0/180] [14335.465100] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14335.493262] LNet: Added LNI 192.168.203.127@tcp32 [8/256/0/180] [14341.863718] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14341.903604] LNet: Added LNI 192.168.203.127@tcp33 [8/256/0/180] [14347.569944] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14347.593329] LNet: Added LNI 192.168.203.127@tcp34 [8/256/0/180] [14353.600760] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14353.629865] LNet: Added LNI 192.168.203.127@tcp35 [8/256/0/180] [14360.032974] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14360.053906] LNet: Added LNI 192.168.203.127@tcp36 [8/256/0/180] [14366.349434] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14366.388249] LNet: Added LNI 192.168.203.127@tcp37 [8/256/0/180] [14373.923912] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14373.972107] LNet: Added LNI 192.168.203.127@tcp38 [8/256/0/180] [14381.100965] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14381.128432] LNet: Added LNI 192.168.203.127@tcp39 [8/256/0/180] [14390.069760] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14390.108339] LNet: Added LNI 192.168.203.127@tcp40 [8/256/0/180] [14398.917581] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14398.954458] LNet: Added LNI 192.168.203.127@tcp41 [8/256/0/180] [14406.042649] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14406.072654] LNet: Added LNI 192.168.203.127@tcp42 [8/256/0/180] [14413.635131] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14413.657618] LNet: Added LNI 192.168.203.127@tcp43 [8/256/0/180] [14423.652786] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14423.695555] LNet: Added LNI 192.168.203.127@tcp44 [8/256/0/180] [14431.910581] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14431.937992] LNet: Added LNI 192.168.203.127@tcp45 [8/256/0/180] [14439.444397] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14439.467302] LNet: Added LNI 192.168.203.127@tcp46 [8/256/0/180] [14447.891878] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14447.925209] LNet: Added LNI 192.168.203.127@tcp47 [8/256/0/180] [14455.903986] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14455.929157] LNet: Added LNI 192.168.203.127@tcp48 [8/256/0/180] [14464.305447] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14464.328390] LNet: Added LNI 192.168.203.127@tcp49 [8/256/0/180] [14472.969149] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14472.994039] LNet: Added LNI 192.168.203.127@tcp50 [8/256/0/180] [14491.620512] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14491.633141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14491.637782] Lustre: Skipped 2 previous similar messages [14491.661693] Lustre: Skipped 2 previous similar messages [14496.738183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14496.754563] Lustre: Skipped 2 previous similar messages [14497.406803] Lustre: server umount lustre-MDT0000 complete [14501.350035] LustreError: 472920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789462026 with bad export cookie 9180317332213781584 [14501.352187] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14501.370427] LustreError: 472920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14501.733586] Lustre: server umount lustre-MDT0001 complete [14517.223401] Lustre: 472203:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789462026/real 1789462026] req@000000000ad57595 x1876386547662208/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789462042 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14517.255984] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14519.811745] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [14522.341614] Lustre: 472204:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789462031/real 1789462031] req@00000000e84d0d05 x1876386547662336/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789462047 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14527.455191] Lustre: 472202:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789462036/real 1789462036] req@000000002196eff9 x1876386547662592/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789462052 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14528.714075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14532.639469] Lustre: 472202:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789462041/real 1789462041] req@00000000596a1bf7 x1876386547662720/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789462057 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14547.039851] LustreError: 472200:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000e84d0d05 x1876386547663488/t0(0) o250->MGC192.168.203.127@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14547.458683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14552.387706] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [14552.549115] LustreError: 474892:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14552.563824] LustreError: 474892:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [14557.668092] LustreError: 474893:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14560.815484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14561.245695] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14562.285602] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14565.650487] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [14572.579995] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14573.580700] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14579.163967] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14598.117667] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14598.128843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14598.134639] Lustre: Skipped 1 previous similar message [14603.233162] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14603.540481] Lustre: server umount lustre-OST0000 complete [14607.333274] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14607.343895] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14607.354436] Lustre: Skipped 1 previous similar message [14607.363941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14607.366743] Lustre: Skipped 1 previous similar message [14613.476027] LustreError: 499141:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14613.497838] LustreError: 499141:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [14613.552744] Lustre: server umount lustre-MDT0000 complete [14617.051685] LustreError: 472921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789462142 with bad export cookie 9180317332213784048 [14617.053465] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14617.067184] LustreError: 472921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14617.436826] Lustre: server umount lustre-MDT0001 complete [14628.720416] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [14631.470693] Key type lgssc unregistered [14631.982633] LNet: 501936:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14632.000618] LNetError: 501936:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14632.018312] LNet: Removed LNI 192.168.203.127@tcp [14632.026158] LNet: Removed LNI 192.168.203.127@tcp10 [14632.030234] LNet: Removed LNI 192.168.203.127@tcp11 [14632.036610] LNet: Removed LNI 192.168.203.127@tcp12 [14632.040651] LNet: Removed LNI 192.168.203.127@tcp13 [14632.044596] LNet: Removed LNI 192.168.203.127@tcp14 [14632.047144] LNet: Removed LNI 192.168.203.127@tcp15 [14632.049276] LNet: Removed LNI 192.168.203.127@tcp16 [14632.051770] LNet: Removed LNI 192.168.203.127@tcp17 [14632.054657] LNet: Removed LNI 192.168.203.127@tcp18 [14632.057804] LNet: Removed LNI 192.168.203.127@tcp19 [14632.061223] LNet: Removed LNI 192.168.203.127@tcp20 [14632.066465] LNet: Removed LNI 192.168.203.127@tcp21 [14632.071338] LNet: Removed LNI 192.168.203.127@tcp22 [14632.083750] LNet: Removed LNI 192.168.203.127@tcp23 [14632.085814] LNet: Removed LNI 192.168.203.127@tcp24 [14632.088917] LNet: Removed LNI 192.168.203.127@tcp25 [14632.091945] LNet: Removed LNI 192.168.203.127@tcp26 [14632.096255] LNet: Removed LNI 192.168.203.127@tcp27 [14632.098569] LNet: Removed LNI 192.168.203.127@tcp28 [14632.100764] LNet: Removed LNI 192.168.203.127@tcp29 [14632.103562] LNet: Removed LNI 192.168.203.127@tcp30 [14632.105898] LNet: Removed LNI 192.168.203.127@tcp31 [14632.108670] LNet: Removed LNI 192.168.203.127@tcp32 [14632.111155] LNet: Removed LNI 192.168.203.127@tcp33 [14632.113460] LNet: Removed LNI 192.168.203.127@tcp34 [14632.116268] LNet: Removed LNI 192.168.203.127@tcp35 [14632.118517] LNet: Removed LNI 192.168.203.127@tcp36 [14632.121214] LNet: Removed LNI 192.168.203.127@tcp37 [14632.123912] LNet: Removed LNI 192.168.203.127@tcp38 [14632.126481] LNet: Removed LNI 192.168.203.127@tcp39 [14632.129097] LNet: Removed LNI 192.168.203.127@tcp40 [14632.132138] LNet: Removed LNI 192.168.203.127@tcp41 [14632.134462] LNet: Removed LNI 192.168.203.127@tcp42 [14632.138881] LNet: Removed LNI 192.168.203.127@tcp43 [14632.140986] LNet: Removed LNI 192.168.203.127@tcp44 [14632.144170] LNet: Removed LNI 192.168.203.127@tcp45 [14632.147254] LNet: Removed LNI 192.168.203.127@tcp46 [14632.149956] LNet: Removed LNI 192.168.203.127@tcp47 [14632.152540] LNet: Removed LNI 192.168.203.127@tcp48 [14632.155242] LNet: Removed LNI 192.168.203.127@tcp49 [14632.164572] LNet: Removed LNI 192.168.203.127@tcp50 [14632.959991] Key type .llcrypt unregistered [14632.961832] Key type ._llcrypt unregistered [14652.927282] Key type ._llcrypt registered [14652.930507] Key type .llcrypt registered [14653.049624] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [14670.084738] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [14670.889211] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14670.902827] alg: No test for adler32 (adler32-zlib) [14672.023277] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [14672.282098] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [14674.007207] Key type lgssc registered [14675.348930] Lustre: Echo OBD driver; http://www.lustre.org/ [14708.127300] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14708.151925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14709.585329] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14709.637729] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14709.768647] Lustre: lustre-MDT0000: new disk, initializing [14709.903605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14709.928077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14714.317665] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [14724.638623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14724.813937] Lustre: 505129:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14724.888860] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14724.893262] Lustre: Skipped 1 previous similar message [14725.030940] Lustre: lustre-MDT0001: new disk, initializing [14725.184237] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14725.237837] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14725.254262] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14730.662356] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [14741.457452] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14748.349906] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14756.322182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14756.527476] Lustre: *** cfs_fail_loc=90f, val=0*** [14756.612759] Lustre: lustre-OST0000: new disk, initializing [14756.616252] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14756.624340] Lustre: 507146:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14756.725374] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14763.345436] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [14766.722955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14766.730627] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14766.863853] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14772.834727] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14782.432449] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14782.444337] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14782.455473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14782.460157] Lustre: Skipped 1 previous similar message [14786.669142] Lustre: server umount lustre-OST0000 complete [14791.653952] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14791.662146] LustreError: Skipped 1 previous similar message [14791.669181] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14791.690283] Lustre: Skipped 1 previous similar message [14791.706710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14792.674972] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14796.582586] Lustre: server umount lustre-MDT0000 complete [14800.034963] LustreError: 505120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789462325 with bad export cookie 1830443255996594232 [14800.049527] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14800.056980] LustreError: 505120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14800.365565] Lustre: server umount lustre-MDT0001 complete [14811.769815] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [14814.960643] Key type lgssc unregistered [14815.402642] LNet: 509052:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14815.422483] LNetError: 509052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14815.497467] LNet: Removed LNI 192.168.203.127@tcp [14816.862970] Key type .llcrypt unregistered [14816.864712] Key type ._llcrypt unregistered [14839.328719] Key type ._llcrypt registered [14839.332094] Key type .llcrypt registered [14839.475770] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [14840.725838] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14840.740372] alg: No test for adler32 (adler32-zlib) [14841.744331] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [14842.044336] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [14843.831195] Key type lgssc registered [14845.247220] Lustre: Echo OBD driver; http://www.lustre.org/ [14854.321299] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14854.351173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14855.828504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14860.319407] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [14865.376669] LustreError: 510320:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14865.415271] LustreError: 510320:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [14868.928482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14869.254531] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14874.093273] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [14881.468505] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14886.857644] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14893.047359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14893.405584] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14899.951180] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [14906.969236] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14917.566258] loop0: detected capacity change from 0 to 204800000 [14917.782314] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14917.802139] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14917.823427] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14924.123804] loop0: detected capacity change from 0 to 204800000 [14924.341099] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14924.368080] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14924.397089] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14929.746808] loop0: detected capacity change from 0 to 204800000 [14935.433030] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14935.572486] Lustre: fs15246-OST0000: new disk, initializing [14935.577838] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [14935.583855] Lustre: 513912:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14935.630768] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14941.414249] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [14950.673116] loop1: detected capacity change from 0 to 204800000 [14955.789656] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14955.874851] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [14955.926931] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [14956.022604] Lustre: fs15246-MDT0000: new disk, initializing [14956.180611] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14956.210218] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14956.312696] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [14957.344852] Lustre: 515036:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000001c6f1598 x1876387290984832/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789462498 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [14960.493415] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [14966.445520] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14966.451962] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14966.465782] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14980.069574] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14980.078670] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [14985.191494] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [14985.887523] Lustre: server umount fs15246-MDT0000 complete [14990.311445] LustreError: 510317:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14995.047933] Lustre: server umount fs15246-OST0000 complete [15007.201864] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15007.210164] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15007.241034] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15008.224229] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15008.229532] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15010.787411] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15010.791609] Lustre: Skipped 2 previous similar messages [15012.282977] Lustre: server umount lustre-OST0000 complete [15015.906178] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15015.909561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15015.926131] Lustre: Skipped 1 previous similar message [15023.750759] Lustre: server umount lustre-MDT0000 complete [15026.145951] LustreError: 510321:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15026.175168] LustreError: 510321:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [15026.853239] LustreError: 510301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789462551 with bad export cookie 16953607747737367349 [15026.854814] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15026.866062] LustreError: 510301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15027.149894] Lustre: server umount lustre-MDT0001 complete [15036.152423] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [15038.573942] Key type lgssc unregistered [15038.843248] LNet: 518130:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15038.851476] LNetError: 518130:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15038.874676] LNet: Removed LNI 192.168.203.127@tcp [15039.611633] Key type .llcrypt unregistered [15039.614991] Key type ._llcrypt unregistered [15057.908557] Key type ._llcrypt registered [15057.911478] Key type .llcrypt registered [15058.041683] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [15059.422538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15059.459886] alg: No test for adler32 (adler32-zlib) [15060.563472] Lustre: Lustre: Build Version: 2.17.58_40_gc6c2479 [15060.816215] LNet: Added LNI 192.168.203.127@tcp [8/256/0/180] [15062.495764] Key type lgssc registered [15063.636448] Lustre: Echo OBD driver; http://www.lustre.org/ [15102.136397] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [15112.110330] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [15149.052233] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15149.085423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15150.284824] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15150.325254] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15150.389389] Lustre: lustre-MDT0000: new disk, initializing [15150.465232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15150.482422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15154.618601] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [15163.514354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15163.610546] Lustre: 522594:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15163.662097] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15163.666224] Lustre: Skipped 1 previous similar message [15163.728477] Lustre: lustre-MDT0001: new disk, initializing [15163.841909] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15163.875258] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15163.895930] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15168.005215] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [15175.788551] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15180.463552] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15186.085222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15186.335124] Lustre: lustre-OST0000: new disk, initializing [15186.341908] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15186.346747] Lustre: 524565:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15186.421885] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15191.815939] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_default_debug -1 all [15196.717037] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15196.738469] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15196.841318] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15200.438774] Lustre: DEBUG MARKER: oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15204.103596] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15204.276148] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15208.271757] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15208.473640] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15212.514803] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15212.523263] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15212.537743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15212.545768] Lustre: Skipped 1 previous similar message [15216.303392] Lustre: server umount lustre-OST0000 complete [15220.198611] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15220.209960] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15220.233020] Lustre: Skipped 1 previous similar message [15220.238788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15222.758821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15222.765451] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15225.936516] Lustre: server umount lustre-MDT0000 complete [15229.027076] LustreError: 522588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789462754 with bad export cookie 17285345422603188259 [15229.032904] LustreError: MGC192.168.203.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15229.036866] LustreError: 522588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15229.339911] Lustre: server umount lustre-MDT0001 complete [15251.233459] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing set_hostid [15262.041326] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing load_modules_local [15297.651532] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15038 sec ========= 05:00:20 (1789462820) [15300.245942] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 05:00:23 (1789462823) === [15304.019668] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 05:00:27 (1789462827) === [15346.347734] Lustre: DEBUG MARKER: oleg327-server.virtnet: executing unload_modules_local [15349.436182] Key type lgssc unregistered [15349.749268] LNet: 531218:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15349.761365] LNetError: 531218:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15349.802979] LNet: Removed LNI 192.168.203.127@tcp [15350.965809] Key type .llcrypt unregistered [15350.970589] Key type ._llcrypt unregistered