[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 495428139 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003335] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.007768] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009011] pid_max: default: 32768 minimum: 301 [ 0.010130] LSM: Security Framework initializing [ 0.011051] Yama: becoming mindful. [ 0.012037] SELinux: Initializing. [ 0.013066] *** VALIDATE selinux *** [ 0.021805] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026328] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027143] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028104] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029105] *** VALIDATE tmpfs *** [ 0.031157] *** VALIDATE proc *** [ 0.032230] *** VALIDATE cgroup *** [ 0.033010] *** VALIDATE cgroup2 *** [ 0.034261] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036041] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038027] Spectre V2 : User space: Vulnerable [ 0.039009] Speculative Store Bypass: Vulnerable [ 0.042012] debug: unmapping init [mem 0xffffffff88259000-0xffffffff88260fff] [ 0.044185] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045675] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046023] ... version: 2 [ 0.047014] ... bit width: 48 [ 0.048016] ... generic registers: 4 [ 0.049014] ... value mask: 0000ffffffffffff [ 0.050014] ... max period: 00007fffffffffff [ 0.051016] ... fixed-purpose events: 3 [ 0.052014] ... event mask: 000000070000000f [ 0.053299] rcu: Hierarchical SRCU implementation. [ 0.055443] smp: Bringing up secondary CPUs ... [ 0.056584] x86: Booting SMP configuration: [ 0.057027] .... node #0, CPUs: #1 #2 #3 [ 0.060391] smp: Brought up 1 node, 4 CPUs [ 0.062012] smpboot: Max logical packages: 1 [ 0.063017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.138784] node 0 deferred pages initialised in 73ms [ 0.143132] devtmpfs: initialized [ 0.144255] x86/mm: Memory block size: 128MB [ 0.147900] gcov: version magic: 0x41383552 [ 0.151301] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.155075] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.158258] pinctrl core: initialized pinctrl subsystem [ 0.161211] [ 0.161746] ************************************************************* [ 0.163011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.166012] ** ** [ 0.168011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.170013] ** ** [ 0.173013] ** This means that this kernel is built to expose internal ** [ 0.175017] ** IOMMU data structures, which may compromise security on ** [ 0.178012] ** your system. ** [ 0.180011] ** ** [ 0.183012] ** If you see this message and you are not debugging the ** [ 0.186022] ** kernel, report this immediately to your vendor! ** [ 0.188013] ** ** [ 0.190012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.192011] ************************************************************* [ 0.194684] NET: Registered protocol family 16 [ 0.197610] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.201089] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.204206] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.208138] cpuidle: using governor menu [ 0.209758] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.211602] PCI: Using configuration type 1 for base access [ 0.214157] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.224137] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.229049] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.232163] cryptd: max_cpu_qlen set to 1000 [ 0.235287] ACPI: Added _OSI(Module Device) [ 0.237015] ACPI: Added _OSI(Processor Device) [ 0.239013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.241014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.246330] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.254395] ACPI: Interpreter enabled [ 0.255112] ACPI: PM: (supports S0 S3 S4 S5) [ 0.256011] ACPI: Using IOAPIC for interrupt routing [ 0.257259] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.260397] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.271860] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.273125] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.276017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.278072] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.283449] acpiphp: Slot [2] registered [ 0.285222] acpiphp: Slot [5] registered [ 0.286130] acpiphp: Slot [6] registered [ 0.287127] acpiphp: Slot [7] registered [ 0.289135] acpiphp: Slot [8] registered [ 0.290082] acpiphp: Slot [9] registered [ 0.291134] acpiphp: Slot [10] registered [ 0.292252] acpiphp: Slot [3] registered [ 0.294087] acpiphp: Slot [4] registered [ 0.295083] acpiphp: Slot [11] registered [ 0.296109] acpiphp: Slot [12] registered [ 0.298096] acpiphp: Slot [13] registered [ 0.299077] acpiphp: Slot [14] registered [ 0.300159] acpiphp: Slot [15] registered [ 0.301109] acpiphp: Slot [16] registered [ 0.302085] acpiphp: Slot [17] registered [ 0.304138] acpiphp: Slot [18] registered [ 0.305153] acpiphp: Slot [19] registered [ 0.307179] acpiphp: Slot [20] registered [ 0.309142] acpiphp: Slot [21] registered [ 0.311107] acpiphp: Slot [22] registered [ 0.313097] acpiphp: Slot [23] registered [ 0.314092] acpiphp: Slot [24] registered [ 0.315101] acpiphp: Slot [25] registered [ 0.317112] acpiphp: Slot [26] registered [ 0.318113] acpiphp: Slot [27] registered [ 0.320104] acpiphp: Slot [28] registered [ 0.322102] acpiphp: Slot [29] registered [ 0.323106] acpiphp: Slot [30] registered [ 0.325102] acpiphp: Slot [31] registered [ 0.327095] PCI host bridge to bus 0000:00 [ 0.329039] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.332041] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.334033] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.337035] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.340039] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.344040] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.345196] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.349314] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.352589] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.363016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.368033] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.371032] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.373017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.376021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.379482] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.382115] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.384065] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.387976] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.393023] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.410021] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.414020] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.420000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.429013] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.438013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.455024] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.464853] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.472021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.483021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.513023] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.530000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.539015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.544017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.569016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.582599] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.591025] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.598024] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.619026] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.630912] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.637027] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.643022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.658021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.667907] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.677023] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.684023] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.711026] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.722864] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.725551] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.730201] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.733367] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.735300] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.741126] iommu: Default domain type: Passthrough [ 0.742489] SCSI subsystem initialized [ 0.744204] ACPI: bus type USB registered [ 0.746312] usbcore: registered new interface driver usbfs [ 0.748094] usbcore: registered new interface driver hub [ 0.749106] usbcore: registered new device driver usb [ 0.751194] pps_core: LinuxPPS API ver. 1 registered [ 0.753017] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.756253] PTP clock support registered [ 0.758055] EDAC MC: Ver: 3.0.0 [ 0.760211] PCI: Using ACPI for IRQ routing [ 0.762563] NetLabel: Initializing [ 0.764017] NetLabel: domain hash size = 128 [ 0.766015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.769098] NetLabel: unlabeled traffic allowed by default [ 0.772143] vgaarb: loaded [ 0.774277] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.775011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.779323] clocksource: Switched to clocksource kvm-clock [ 0.884559] VFS: Disk quotas dquot_6.6.0 [ 0.886430] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.889398] *** VALIDATE ramfs *** [ 0.890903] *** VALIDATE hugetlbfs *** [ 0.892617] pnp: PnP ACPI init [ 0.895051] pnp: PnP ACPI: found 6 devices [ 0.921972] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.925747] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.928682] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.931180] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.933826] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.936577] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.938769] NET: Registered protocol family 2 [ 0.941203] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.945753] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.948933] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.954822] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.958426] TCP: Hash tables configured (established 65536 bind 65536) [ 0.961439] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.964800] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.968139] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.971020] NET: Registered protocol family 1 [ 0.973557] RPC: Registered named UNIX socket transport module. [ 0.975867] RPC: Registered udp transport module. [ 0.977842] RPC: Registered tcp transport module. [ 0.979581] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.982317] NET: Registered protocol family 44 [ 0.983935] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.985984] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.988424] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.991147] PCI: CLS 0 bytes, default 64 [ 0.992947] Unpacking initramfs... [ 2.463112] debug: unmapping init [mem 0xffff9afbbcc54000-0xffff9afbbffbffff] [ 2.467173] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.469782] software IO TLB: mapped [mem 0x00000000a0e00000-0x00000000a4e00000] (64MB) [ 2.472990] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.012526] Initialise system trusted keyrings [ 3.017130] Key type blacklist registered [ 3.019069] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.028480] zbud: loaded [ 3.031699] *** VALIDATE nfs *** [ 3.032982] *** VALIDATE nfs4 *** [ 3.034701] pstore: using deflate compression [ 3.038281] Platform Keyring initialized [ 3.170683] NET: Registered protocol family 38 [ 3.172787] Key type asymmetric registered [ 3.174465] Asymmetric key parser 'x509' registered [ 3.176469] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.179311] io scheduler mq-deadline registered [ 3.180941] io scheduler kyber registered [ 3.183081] io scheduler bfq registered [ 3.184840] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.187958] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.190932] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.193103] ACPI: Power Button [PWRF] [ 3.197560] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.205610] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.223303] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.234678] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.251492] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.278085] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.310090] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.315533] Non-volatile memory driver v1.3 [ 3.317317] Linux agpgart interface v0.103 [ 3.356771] virtio_blk virtio1: [vda] 146136 512-byte logical blocks (74.8 MB/71.4 MiB) [ 3.361239] vda: detected capacity change from 0 to 74821632 [ 3.381257] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.384488] vdb: detected capacity change from 0 to 1073741824 [ 3.412867] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.417587] vdc: detected capacity change from 0 to 2621440000 [ 3.442055] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.445868] vdd: detected capacity change from 0 to 2621440000 [ 3.468520] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.471586] vde: detected capacity change from 0 to 4294967296 [ 3.493901] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.497205] vdf: detected capacity change from 0 to 4294967296 [ 3.512539] libphy: Fixed MDIO Bus: probed [ 3.519389] usbcore: registered new interface driver usbserial_generic [ 3.521937] usbserial: USB Serial support registered for generic [ 3.524813] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.530333] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.532436] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.535375] mousedev: PS/2 mouse device common for all mice [ 3.543372] rtc_cmos 00:05: RTC can wake from S4 [ 3.543427] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.550062] rtc_cmos 00:05: registered as rtc0 [ 3.552121] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.554292] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.556080] intel_pstate: CPU model not supported [ 3.560498] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.566742] hid: raw HID events driver (C) Jiri Kosina [ 3.568755] usbcore: registered new interface driver usbhid [ 3.570554] usbhid: USB HID core driver [ 3.572179] drop_monitor: Initializing network drop monitor service [ 3.574612] Initializing XFRM netlink socket [ 3.576542] NET: Registered protocol family 10 [ 3.579725] Segment Routing with IPv6 [ 3.581281] NET: Registered protocol family 17 [ 3.583432] mpls_gso: MPLS GSO support [ 3.590344] RAS: Correctable Errors collector initialized. [ 3.592299] AVX version of gcm_enc/dec engaged. [ 3.593721] AES CTR mode by8 optimization enabled [ 3.699344] sched_clock: Marking stable (3699316056, 0)->(4605568899, -906252843) [ 3.704613] registered taskstats version 1 [ 3.707426] Loading compiled-in X.509 certificates [ 3.710321] zswap: loaded using pool lzo/zbud [ 3.737314] Key type big_key registered [ 3.753524] Key type encrypted registered [ 3.755456] ima: No TPM chip found, activating TPM-bypass! [ 3.756959] ima: Allocated hash algorithm: sha1 [ 3.758590] ima: No architecture policies found [ 3.760301] evm: Initialising EVM extended attributes: [ 3.761867] evm: security.selinux [ 3.763016] evm: security.ima [ 3.763855] evm: security.capability [ 3.764967] evm: HMAC attrs: 0x1 [ 3.767402] rtc_cmos 00:05: setting system clock to 2026-08-23 04:55:49 UTC (1787460949) [ 3.778336] debug: unmapping init [mem 0xffffffff89203000-0xffffffff893fffff] [ 3.781500] debug: unmapping init [mem 0xffffffff87f82000-0xffffffff88258fff] [ 3.790139] Write protecting the kernel read-only data: 28672k [ 3.794247] debug: unmapping init [mem 0xffffffff86603000-0xffffffff867fffff] [ 3.796551] debug: unmapping init [mem 0xffffffff86f14000-0xffffffff86ffffff] [ 3.829087] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.836834] systemd[1]: Detected virtualization kvm. [ 3.839849] systemd[1]: Detected architecture x86-64. [ 3.842635] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.871168] systemd[1]: No hostname configured. [ 3.873702] systemd[1]: Set hostname to . [ 3.876713] random: systemd: uninitialized urandom read (16 bytes read) [ 3.880425] systemd[1]: Initializing machine ID from random generator. [ 3.935915] random: ln: uninitialized urandom read (6 bytes read) [ 4.020815] random: systemd: uninitialized urandom read (16 bytes read) [ 4.025218] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 4.034424] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ 4.042332] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.648912] device-mapper: uevent: version 1.0.3 [ 4.650721] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.325481] virtio_net virtio0 ens2: renamed from eth0 [ 5.341032] random: fast init done [ 5.408564] scsi host0: ata_piix [ 5.420235] scsi host1: ata_piix [ 5.450429] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.452837] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.084820] dracut-initqueue[578]: RTNETLINK answers: File exists [ 10.271752] random: crng init done [ 10.272990] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.605065] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.691782] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.945402] SELinux: Disabled at runtime. [ 12.004618] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.013608] systemd[1]: Detected virtualization kvm. [ 12.015706] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.497488] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.500928] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.508665] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.512660] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.515693] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.523558] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.531716] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... Mounting Kernel Debug File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting Huge Pages File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. [ 12.646291] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue File System... Starting Apply Kernel Variables... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 12.908094] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.206663] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.233941] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.312564] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.329318] EDAC sbridge: Ver: 1.1.2 [ 14.883668] Key type dns_resolver registered [ 15.221781] NFS: Registering the id_resolver key type [ 15.225610] Key type id_resolver registered [ 15.228651] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg625-server login: [ 41.753260] libcfs: loading out-of-tree module taints kernel. [ 41.831720] Key type ._llcrypt registered [ 41.834306] Key type .llcrypt registered [ 41.971705] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_hostid [ 61.181415] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing load_modules_local [ 62.866422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 62.878343] alg: No test for adler32 (adler32-zlib) [ 63.156094] hrtimer: interrupt took 4003175 ns [ 64.337192] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 65.406579] LNet: Added LNI 192.168.206.125@tcp [8/256/0/180] [ 67.232212] Key type lgssc registered [ 69.465746] Lustre: Echo OBD driver; http://www.lustre.org/ [ 86.094376] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 135.351935] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing load_modules_local [ 149.863129] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 149.905109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 151.056489] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 151.100898] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 151.196343] Lustre: lustre-MDT0000: new disk, initializing [ 151.363936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 151.382603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 156.106718] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 172.322301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 172.434719] Lustre: 6537:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 172.468975] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 172.472737] Lustre: Skipped 1 previous similar message [ 172.555188] Lustre: lustre-MDT0001: new disk, initializing [ 172.614860] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 172.637364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 172.649591] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 177.500577] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 182.975560] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 193.426311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 193.648915] Lustre: lustre-OST0000: new disk, initializing [ 193.652400] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 193.665070] Lustre: 8473:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 193.734065] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 199.233212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 199.244553] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 199.340517] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 200.354560] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 214.232547] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 214.345594] Lustre: lustre-OST0001: new disk, initializing [ 214.350700] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 214.359508] Lustre: 9547:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 214.413096] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 220.230779] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 220.245543] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 220.351401] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 220.408403] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 232.391354] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 239.647277] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 246.665630] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing check_logdir /tmp/testlogs/ [ 252.148801] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing yml_node [ 257.003687] Lustre: DEBUG MARKER: Client: 2.17.57.65 [ 259.893159] Lustre: DEBUG MARKER: MDS: 2.17.57.65 [ 263.112193] Lustre: DEBUG MARKER: OSS: 2.17.57.65 [ 264.728923] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Aug 23 01:00:08 EDT 2026 [ 283.548307] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 285.144952] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 302.560363] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 302.575483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 302.580351] Lustre: Skipped 1 previous similar message [ 302.597396] Lustre: Skipped 2 previous similar messages [ 307.680414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 307.700089] Lustre: Skipped 4 previous similar messages [ 312.805550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 315.359110] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 315.703929] Lustre: server umount lustre-MDT0000 complete [ 319.677300] LustreError: 6528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787461265 with bad export cookie 16696667818156317946 [ 319.679535] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 319.695079] LustreError: 6528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 323.044460] LustreError: 6542:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 323.044827] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 323.047025] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 323.047033] Lustre: Skipped 3 previous similar messages [ 323.068444] LustreError: 6542:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 323.088623] Lustre: Skipped 3 previous similar messages [ 324.064344] LustreError: 12287:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 324.082201] LustreError: 12287:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 329.193630] LustreError: 6542:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 329.193634] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 329.220195] Lustre: Skipped 4 previous similar messages [ 334.303914] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 334.800191] Lustre: server umount lustre-MDT0001 complete [ 339.073442] Lustre: server umount lustre-OST0000 complete [ 343.262792] Lustre: server umount lustre-OST0001 complete [ 350.927784] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_hostid [ 360.575788] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing load_modules_local [ 402.783483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 403.017162] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 403.038419] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 403.098483] Lustre: lustre-MDT0000: new disk, initializing [ 403.189075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 403.199486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 407.857690] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 419.778942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 419.998374] Lustre: 15117:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 420.153640] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 420.158409] Lustre: Skipped 1 previous similar message [ 420.253109] Lustre: lustre-MDT0001: new disk, initializing [ 420.428604] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 420.469655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 420.486756] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 425.172416] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 434.657143] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 440.599455] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 448.611205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 449.034258] Lustre: lustre-OST0000: new disk, initializing [ 449.046501] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 449.059131] Lustre: 17088:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 449.192019] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 450.494118] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 450.510553] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 450.684761] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 456.598630] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 468.292550] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 473.475078] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 473.754871] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 478.452476] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 478.690590] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 481.252389] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 481.267161] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 481.286317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 481.295804] Lustre: Skipped 3 previous similar messages [ 487.200429] Lustre: server umount lustre-OST0000 complete [ 492.000555] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 492.013458] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 492.034081] Lustre: Skipped 1 previous similar message [ 496.611537] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 501.728733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 501.733020] Lustre: Skipped 6 previous similar messages [ 506.335313] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 506.566921] Lustre: server umount lustre-MDT0000 complete [ 506.855937] LustreError: 15962:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 506.867657] LustreError: 15962:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 510.310439] LustreError: 15109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787461456 with bad export cookie 16696667818156319479 [ 510.316650] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 510.319737] LustreError: 15109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 510.707312] Lustre: server umount lustre-MDT0001 complete [ 517.341600] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 01:04:20 (1787461460) [ 536.068855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 536.485990] LustreError: 19966:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 536.571037] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 541.004306] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 546.804978] LustreError: 19967:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 546.822171] LustreError: 19967:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 549.623622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 554.632493] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 562.093781] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 567.259866] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 569.194054] Lustre: Failing over lustre-MDT0000 [ 569.412627] Lustre: server umount lustre-MDT0000 complete [ 570.342967] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 570.357980] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 570.374768] LustreError: 19966:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 588.767882] Lustre: 3653:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787461518/real 1787461518] req@ffff9afc34a35180 x1874288717233792/t0(0) o400->MGC192.168.206.125@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787461534 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 588.809154] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 589.046381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 599.011724] LustreError: 3652:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9afb032df100 x1874288717236608/t0(0) o250->MGC192.168.206.125@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 603.963993] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 604.645923] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 604.652518] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 604.681845] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 613.614528] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 617.205216] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 617.207069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 617.220871] Lustre: Skipped 2 previous similar messages [ 623.515473] Lustre: server umount lustre-MDT0000 complete [ 624.613955] LustreError: 19961:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 624.644452] LustreError: 19961:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [ 627.340644] LustreError: 19946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787461573 with bad export cookie 16696667818156320893 [ 627.341780] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 627.349959] LustreError: 19946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 627.683615] Lustre: server umount lustre-MDT0001 complete [ 636.467244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 636.998065] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 637.005408] Lustre: Skipped 2 previous similar messages [ 641.690299] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 651.081903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 656.371932] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 664.240222] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 670.509700] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 673.305054] Lustre: Failing over lustre-MDT0000 [ 673.556290] Lustre: server umount lustre-MDT0000 complete [ 676.386353] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 676.405465] Lustre: Skipped 1 previous similar message [ 691.684679] LustreError: 23303:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 691.706361] LustreError: 23303:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 692.706274] Lustre: 3653:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787461622/real 1787461622] req@ffff9afb02f5e680 x1874288717262720/t0(0) o400->MGC192.168.206.125@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787461638 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 692.769092] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 695.394107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 703.079421] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xe7b67aeaf9e76c27 [ 703.100491] LustreError: 25308:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.206.125@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (ffff9afc186a5400) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 703.127786] Lustre: MGC192.168.206.125@tcp: Connection restored to 0@lo (at 0@lo) [ 703.135288] Lustre: Skipped 1 previous similar message [ 703.330924] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 708.005988] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 708.584636] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 708.610099] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 708.638142] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 719.180925] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 723.936909] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 723.953349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 723.965636] Lustre: Skipped 3 previous similar messages [ 738.274520] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 738.405104] Lustre: server umount lustre-MDT0000 complete [ 742.403177] LustreError: 23285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787461688 with bad export cookie 16696667818156321831 [ 742.412575] LustreError: 23285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 742.414575] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 742.782347] Lustre: server umount lustre-MDT0001 complete [ 752.327648] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 01:08:15 (1787461695) [ 772.195041] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_hostid [ 781.057202] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing load_modules_local [ 828.340958] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing load_modules_local [ 839.865494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 840.050354] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 840.092930] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 840.161220] Lustre: lustre-MDT0000: new disk, initializing [ 840.251573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 840.255099] Lustre: Skipped 2 previous similar messages [ 840.272525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 846.007966] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 858.376407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 858.562282] Lustre: 29684:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 858.944987] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 863.458851] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 874.136504] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 880.211435] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 886.860060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 887.189161] Lustre: 31652:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 889.126812] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 889.188800] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 889.254270] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 889.270966] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 893.411369] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 902.996968] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 910.267495] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 930.272590] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 930.275117] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 930.284346] LustreError: Skipped 1 previous similar message [ 930.307183] Lustre: Skipped 2 previous similar messages [ 930.309423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 930.333938] Lustre: Skipped 7 previous similar messages [ 931.022970] Lustre: server umount lustre-OST0000 complete [ 940.905172] Lustre: server umount lustre-MDT0000 complete [ 944.487947] LustreError: 29676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787461890 with bad export cookie 16696667818156322244 [ 944.491751] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 944.505746] LustreError: 29676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 944.873349] Lustre: server umount lustre-MDT0001 complete [ 956.497710] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing unload_modules_local [ 959.916428] Key type lgssc unregistered [ 960.481527] LNet: 33634:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 960.494127] LNetError: 33634:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 960.514504] LNet: Removed LNI 192.168.206.125@tcp [ 962.064150] Key type .llcrypt unregistered [ 962.065937] Key type ._llcrypt unregistered [ 992.029120] Key type ._llcrypt registered [ 992.034115] Key type .llcrypt registered [ 992.125173] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing load_modules_local [ 992.773244] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 992.909974] alg: No test for adler32 (adler32-zlib) [ 993.995935] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 994.248572] LNet: Added LNI 192.168.206.125@tcp [8/256/0/180] [ 996.031281] Key type lgssc registered [ 997.294900] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1005.726339] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1005.778299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1007.199398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1011.400325] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 1016.298682] LustreError: 35050:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1016.321616] LustreError: 35050:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1020.646523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1020.991879] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1025.992481] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 1034.001145] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1040.095228] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1047.371594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1047.842741] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1054.541066] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 1062.787549] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1065.350627] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 1077.906090] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 01:13:41 (1787462021) [ 1080.014656] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1082.154976] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1088.948751] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 01:13:52 (1787462032) [ 1098.378750] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 01:14:01 (1787462041) [ 1099.411416] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1108.210941] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1108.220888] Lustre: Skipped 2 previous similar messages [ 1110.130358] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 01:14:13 (1787462053) [ 1114.096099] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1114.108406] Lustre: Skipped 3 previous similar messages [ 1133.969881] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1133.977505] Lustre: Skipped 6 previous similar messages [ 1136.742954] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 01:14:40 (1787462080) [ 1139.372256] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1142.683992] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1154.306815] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 01:14:56 (1787462096) [ 1157.049841] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1157.053612] Lustre: Skipped 3 previous similar messages [ 1174.405218] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 01:15:17 (1787462117) [ 1193.440835] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1193.452805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1193.455702] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1193.460073] Lustre: Skipped 1 previous similar message [ 1193.505680] Lustre: Skipped 1 previous similar message [ 1196.819219] Lustre: server umount lustre-OST0000 complete [ 1203.680513] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1203.685477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1203.687524] Lustre: Skipped 1 previous similar message [ 1207.102993] Lustre: server umount lustre-MDT0000 complete [ 1208.802600] LustreError: 35779:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1208.831334] LustreError: 35779:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1210.897196] LustreError: 35030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787462156 with bad export cookie 4604195228329199666 [ 1210.898459] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1210.906077] LustreError: 35030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1211.253177] Lustre: server umount lustre-MDT0001 complete [ 1222.611978] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing unload_modules_local [ 1225.076402] Key type lgssc unregistered [ 1225.370218] LNet: 42914:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1225.380137] LNetError: 42914:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1225.405518] LNet: Removed LNI 192.168.206.125@tcp [ 1226.313923] Key type .llcrypt unregistered [ 1226.316289] Key type ._llcrypt unregistered [ 1243.882685] Key type ._llcrypt registered [ 1243.884735] Key type .llcrypt registered [ 1244.003779] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_hostid [ 1260.436719] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing load_modules_local [ 1261.897673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1261.998775] alg: No test for adler32 (adler32-zlib) [ 1263.137747] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 1263.449332] LNet: Added LNI 192.168.206.125@tcp [8/256/0/180] [ 1265.143425] Key type lgssc registered [ 1266.432371] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1306.770146] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1306.800522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1308.149288] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1308.191331] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1308.318374] Lustre: lustre-MDT0000: new disk, initializing [ 1308.469426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1308.505099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1313.302265] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 1324.589240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1324.732672] Lustre: 46068:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1324.783260] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1324.787352] Lustre: Skipped 1 previous similar message [ 1324.863787] Lustre: lustre-MDT0001: new disk, initializing [ 1324.960399] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1325.021245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1325.039914] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1329.558184] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 1339.816824] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1346.442162] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1354.105876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1354.371532] Lustre: lustre-OST0000: new disk, initializing [ 1354.374214] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1354.378675] Lustre: 48036:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1354.467250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1360.494711] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 1360.968663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1360.978476] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1361.113467] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1369.859850] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1375.000305] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1375.293247] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1380.791771] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1381.018543] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1386.990419] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1386.995841] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1387.003157] LustreError: Skipped 1 previous similar message [ 1387.007923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1387.029527] Lustre: Skipped 1 previous similar message [ 1392.101303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1392.110801] Lustre: Skipped 1 previous similar message [ 1397.215310] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1397.219242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1397.232717] Lustre: Skipped 1 previous similar message [ 1397.374844] Lustre: server umount lustre-OST0000 complete [ 1401.824226] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1401.828160] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1401.842775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1401.847458] Lustre: Skipped 1 previous similar message [ 1407.612474] Lustre: server umount lustre-MDT0000 complete [ 1411.735170] LustreError: 46862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787462357 with bad export cookie 8611580156703367142 [ 1411.740130] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1411.743478] LustreError: 46862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1412.146883] Lustre: server umount lustre-MDT0001 complete [ 1421.001919] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 01:19:24 (1787462364) [ 1428.256939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1428.892432] LustreError: 50496:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1428.918358] LustreError: 50496:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1428.992433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1433.820818] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 1434.085158] LustreError: 50497:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1439.205441] LustreError: 50496:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1442.991825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1443.506853] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1448.321135] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 1455.994452] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1461.910709] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1468.718996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1469.074855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1476.254391] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 1483.635806] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1491.760298] LustreError: 53025:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1492.727477] Lustre: Modifying parameter general.timeout=1 in log params [ 1494.295538] Lustre: Modifying parameter general.timeout=3 in log params [ 1494.303086] Lustre: Skipped 1 previous similar message [ 1496.573495] Lustre: Modifying parameter general.timeout=6 in log params [ 1496.580990] Lustre: Skipped 2 previous similar messages [ 1500.998367] Lustre: Modifying parameter general.timeout=12 in log params [ 1501.001730] Lustre: Skipped 5 previous similar messages [ 1509.454789] Lustre: Modifying parameter general.timeout=21 in log params [ 1509.461251] Lustre: Skipped 8 previous similar messages [ 1525.770861] Lustre: Modifying parameter general.timeout=36 in log params [ 1525.779272] Lustre: Skipped 14 previous similar messages [ 1558.368784] Lustre: Modifying parameter general.timeout=70 in log params [ 1558.370996] Lustre: Skipped 33 previous similar messages [ 1623.046532] Lustre: Modifying parameter general.timeout=129 in log params [ 1623.054394] Lustre: Skipped 58 previous similar messages [ 1637.606643] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 01:23:00 (1787462580) [ 1641.158283] Lustre: 59976:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1643.203730] Lustre: 60075:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1645.209168] Lustre: 60188:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1651.167687] Lustre: 60482:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1651.177196] Lustre: 60482:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1680.770529] Lustre: 61859:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1680.819664] LustreError: 61859:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1685.205323] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 01:23:48 (1787462628) [ 1699.308962] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1699.328390] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1699.347298] Lustre: Skipped 1 previous similar message [ 1699.376497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1699.385484] Lustre: Skipped 2 previous similar messages [ 1701.862299] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1705.444486] Lustre: server umount lustre-MDT0000 complete [ 1706.982785] LustreError: 50492:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1707.010493] LustreError: 50492:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1709.538404] LustreError: 50478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787462655 with bad export cookie 8611580156703368024 [ 1709.542964] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1709.552101] LustreError: 50478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1709.844812] Lustre: server umount lustre-MDT0001 complete [ 1724.979570] Lustre: server umount lustre-OST0000 complete [ 1753.277275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1753.652389] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1753.720309] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1753.728172] Lustre: Skipped 23 previous similar messages [ 1754.077882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1759.184392] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 1768.873715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1769.021886] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1769.085400] Lustre: 64393:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1774.646838] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 1782.076795] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1787.609262] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1793.717532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1793.937237] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1794.371860] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1794.382521] Lustre: Skipped 1 previous similar message [ 1800.554433] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 1807.629798] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1809.631737] Lustre: 66959:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1819.109130] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1819.121557] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1819.123284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1819.139810] Lustre: Skipped 2 previous similar messages [ 1819.144081] Lustre: Skipped 3 previous similar messages [ 1823.052574] Lustre: server umount lustre-OST0000 complete [ 1829.354783] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1829.368026] Lustre: Skipped 1 previous similar message [ 1833.351202] Lustre: server umount lustre-MDT0000 complete [ 1834.464771] LustreError: 64404:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1834.494807] LustreError: 64404:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1838.206414] LustreError: 66255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787462783 with bad export cookie 8611580156703371237 [ 1838.220913] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1838.231695] LustreError: 66255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1838.783442] Lustre: server umount lustre-MDT0001 complete [ 1851.998466] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing unload_modules_local [ 1855.140688] Key type lgssc unregistered [ 1855.517914] LNet: 68175:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1855.529090] LNetError: 68175:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1855.564504] LNet: Removed LNI 192.168.206.125@tcp [ 1856.717159] Key type .llcrypt unregistered [ 1856.724531] Key type ._llcrypt unregistered [ 1883.158956] Key type ._llcrypt registered [ 1883.162667] Key type .llcrypt registered [ 1883.249091] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing load_modules_local [ 1884.513092] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1884.558232] alg: No test for adler32 (adler32-zlib) [ 1885.628345] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 1885.859272] LNet: Added LNI 192.168.206.125@tcp [8/256/0/180] [ 1887.520782] Key type lgssc registered [ 1888.845605] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1896.585810] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1896.598310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1898.168950] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1903.232194] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 1907.690932] LustreError: 69590:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1907.712423] LustreError: 69590:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1912.801295] LustreError: 69591:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1915.029480] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1915.977862] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1921.235170] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 1929.351240] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1935.020627] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1941.946969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1942.650313] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1949.305412] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 1957.051926] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1969.656361] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1980.898915] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1980.906525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1980.907686] Lustre: Skipped 1 previous similar message [ 1980.927780] Lustre: Skipped 2 previous similar messages [ 1986.025427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1986.038755] Lustre: Skipped 2 previous similar messages [ 1986.529634] Lustre: server umount lustre-MDT0000 complete [ 1991.144507] LustreError: 69587:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1991.175366] LustreError: 69587:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1991.466746] LustreError: 69570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787462937 with bad export cookie 15136555862132017355 [ 1991.479967] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1991.494512] LustreError: 69570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1991.975199] Lustre: server umount lustre-MDT0001 complete [ 2005.750457] Lustre: server umount lustre-OST0000 complete [ 2037.803862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2038.146883] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2038.206262] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2038.492560] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2044.217268] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 2053.995296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2054.075776] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2054.110349] Lustre: 74464:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2054.597454] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2058.798338] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 2066.500601] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2073.256434] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2081.292698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2081.521609] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2082.161972] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2088.909703] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 2096.927472] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2098.891058] Lustre: 77038:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2100.852458] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2106.848990] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2106.859356] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2106.888320] Lustre: Skipped 1 previous similar message [ 2106.898615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2107.378276] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2112.338441] Lustre: server umount lustre-OST0000 complete [ 2117.601631] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2117.604228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2117.621994] Lustre: Skipped 2 previous similar messages [ 2117.634335] Lustre: Skipped 3 previous similar messages [ 2122.721148] LustreError: 74471:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2122.733427] LustreError: 74471:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2122.809483] Lustre: server umount lustre-MDT0000 complete [ 2126.882516] LustreError: 75209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787463072 with bad export cookie 15136555862132018202 [ 2126.889345] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2126.890391] LustreError: 75209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2127.154924] Lustre: server umount lustre-MDT0001 complete [ 2137.976778] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing unload_modules_local [ 2141.325654] Key type lgssc unregistered [ 2141.635718] LNet: 78254:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2141.639847] LNetError: 78254:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2141.648301] LNet: Removed LNI 192.168.206.125@tcp [ 2142.472803] Key type .llcrypt unregistered [ 2142.476395] Key type ._llcrypt unregistered [ 2167.158820] Key type ._llcrypt registered [ 2167.161686] Key type .llcrypt registered [ 2167.253214] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing load_modules_local [ 2168.355558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2168.401107] alg: No test for adler32 (adler32-zlib) [ 2169.552698] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 2169.796281] LNet: Added LNI 192.168.206.125@tcp [8/256/0/180] [ 2171.480543] Key type lgssc registered [ 2172.688493] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2180.677264] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2180.691434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2182.199719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2185.869857] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 2191.330596] LustreError: 79666:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2191.384292] LustreError: 79666:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2196.180361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2196.452688] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2196.790585] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2203.231611] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 2214.711907] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2222.527977] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2231.005367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2231.589914] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2239.350522] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 2248.321550] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2260.722497] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2261.879991] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2263.156636] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2265.354644] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2265.365276] Lustre: Skipped 1 previous similar message [ 2270.177396] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2270.183121] Lustre: Skipped 3 previous similar messages [ 2279.092738] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2279.097503] Lustre: Skipped 7 previous similar messages [ 2295.185309] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2295.189245] Lustre: Skipped 15 previous similar messages [ 2327.520951] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2327.527951] Lustre: Skipped 33 previous similar messages [ 2369.549806] Lustre: 87794:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2372.796928] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 01:35:15 (1787463315) [ 2374.461586] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2376.175612] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 01:35:19 (1787463319) [ 2385.383871] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2385.390384] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2385.397338] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2385.429164] Lustre: Skipped 1 previous similar message [ 2390.504237] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2390.520299] Lustre: Skipped 2 previous similar messages [ 2395.103280] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2395.351566] Lustre: server umount lustre-OST0000 complete [ 2400.743885] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2400.750811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2400.771818] Lustre: Skipped 1 previous similar message [ 2400.790171] Lustre: Skipped 1 previous similar message [ 2405.858435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2405.862837] Lustre: Skipped 1 previous similar message [ 2406.361630] Lustre: server umount lustre-MDT0000 complete [ 2410.861911] LustreError: 79647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787463356 with bad export cookie 468328907272087046 [ 2410.870146] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2410.881667] LustreError: 79647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2411.423639] Lustre: server umount lustre-MDT0001 complete [ 2428.283927] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing unload_modules_local [ 2432.300921] Key type lgssc unregistered [ 2432.751743] LNet: 89047:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2432.768870] LNetError: 89047:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2432.795828] LNet: Removed LNI 192.168.206.125@tcp [ 2434.006735] Key type .llcrypt unregistered [ 2434.008168] Key type ._llcrypt unregistered [ 2438.358156] Key type ._llcrypt registered [ 2438.362524] Key type .llcrypt registered [ 2438.501156] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2442.908084] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing load_modules [ 2443.504537] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2443.586287] alg: No test for adler32 (adler32-zlib) [ 2444.628803] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 2444.836131] LNet: Added LNI 192.168.206.125@tcp [8/256/0/180] [ 2446.544600] Key type lgssc registered [ 2447.619101] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2449.226476] LustreError: 90249:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2467.667959] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing load_modules_local [ 2471.145732] LustreError: 90249:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2477.411173] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2477.431794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2479.075032] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2484.254821] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 2488.301797] LustreError: 91273:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2488.323497] LustreError: 91273:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2493.309372] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 01:37:16 (1787463436) [ 2493.413882] LustreError: 91274:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2497.905352] LustreError: 91305:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2497.918478] LustreError: 91305:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2497.926140] LustreError: 91305:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 2498.251934] Lustre: server umount lustre-MDT0000 complete [ 2510.752321] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing unload_modules_local [ 2513.668576] Key type lgssc unregistered [ 2514.052767] LNet: 92800:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2514.058882] LNetError: 92800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2514.081606] LNet: Removed LNI 192.168.206.125@tcp [ 2514.876747] Key type .llcrypt unregistered [ 2514.880738] Key type ._llcrypt unregistered [ 2532.138262] Key type ._llcrypt registered [ 2532.142980] Key type .llcrypt registered [ 2532.245601] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing load_modules_local [ 2532.979977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2532.998198] alg: No test for adler32 (adler32-zlib) [ 2534.005309] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 2534.218485] LNet: Added LNI 192.168.206.125@tcp [8/256/0/180] [ 2535.879157] Key type lgssc registered [ 2537.023668] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2544.041440] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2544.065893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2545.589967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2549.770528] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 2555.873815] LustreError: 94019:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2555.888940] LustreError: 94019:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2557.958810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2558.640466] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2563.054283] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 2570.296496] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2575.710309] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2582.248048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2582.802588] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2589.014392] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 2596.375453] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2642.707733] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 01:39:46 (1787463586) [ 2644.341237] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2645.945854] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 01:39:49 (1787463589) [ 2650.603314] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2650.614288] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2650.636941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2652.642175] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2652.653387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2652.657225] Lustre: Skipped 1 previous similar message [ 2652.663032] Lustre: Skipped 1 previous similar message [ 2656.296655] Lustre: server umount lustre-MDT0000 complete [ 2657.770696] LustreError: 94014:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2657.790312] LustreError: 94014:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2660.136798] LustreError: 94000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787463605 with bad export cookie 6418404675573882863 [ 2660.145485] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2660.145773] LustreError: 94000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2660.519507] Lustre: server umount lustre-MDT0001 complete [ 2675.270879] Lustre: server umount lustre-OST0000 complete [ 2686.920604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2687.686092] LustreError: 98312:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2687.839105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2692.135384] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 2693.093638] LustreError: 98313:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2698.211626] LustreError: 98312:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2700.907749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2701.548044] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2705.784962] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 2713.562045] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2719.184436] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2732.581943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2732.788497] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2732.794195] LustreError: 98302:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2732.818860] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2732.830667] LustreError: 100405:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2732.834190] LustreError: 100405:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2732.846058] LustreError: 100405:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2732.952515] Lustre: server umount lustre-OST0000 complete [ 2732.954673] LustreError: 100405:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2741.058441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2741.333997] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2741.339428] LustreError: 98302:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2741.349948] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2741.357774] LustreError: 100867:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2741.366329] LustreError: 100867:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2741.371516] LustreError: 100867:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2741.452847] Lustre: server umount lustre-OST0000 complete [ 2741.457249] LustreError: 100867:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2750.870249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2751.013430] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2751.025610] Lustre: Found index 0 for lustre-OST0000, updating log [ 2751.033639] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2751.132524] Lustre: lustre-OST0000: new disk, initializing [ 2751.139935] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2751.148358] Lustre: 101402:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2751.357988] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2758.303876] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 2762.204965] Lustre: Failing over lustre-OST0000 [ 2762.375702] Lustre: server umount lustre-OST0000 complete [ 2767.861026] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2767.866702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2767.868099] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2767.876510] Lustre: Skipped 1 previous similar message [ 2767.897179] Lustre: Skipped 2 previous similar messages [ 2770.987285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2776.180452] Lustre: server umount lustre-MDT0000 complete [ 2779.106857] LustreError: 98312:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2779.142955] LustreError: 98312:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2779.843789] LustreError: 100424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787463725 with bad export cookie 6418404675573885607 [ 2779.849734] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2779.851374] LustreError: 100424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2780.326527] Lustre: server umount lustre-MDT0001 complete [ 2789.897175] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 01:42:13 (1787463733) [ 2809.930883] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing load_modules_local [ 2819.645225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2819.928974] LustreError: 103747:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2820.017429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2824.513478] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 2832.719811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2838.807170] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 2842.952633] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2853.233331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2854.986627] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2854.989220] Lustre: Skipped 1 previous similar message [ 2855.006329] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2858.085147] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2858.108864] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 2858.178502] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2862.768812] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 2863.596236] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 2863.629948] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2864.683874] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:03 [ 2864.712637] Lustre: lustre-OST0000: Recovery over after 0:06, of 1 clients 1 recovered and 0 were evicted. [ 2864.714725] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2864.725957] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2871.253883] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2871.301492] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2871.362545] Lustre: lustre-OST0001: new disk, initializing [ 2871.365490] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2871.375877] Lustre: 106069:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2877.019022] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2877.037071] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2877.123048] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2877.850322] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 2885.205569] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2895.420505] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2900.840417] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.25@tcp (stopping) [ 2900.846235] Lustre: Skipped 3 previous similar messages [ 2903.020745] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2906.950213] Lustre: server umount lustre-MDT0001 complete [ 2908.144097] LustreError: 104750:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2908.186484] LustreError: 104750:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2920.574792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2920.832100] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2920.836758] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2920.874676] Lustre: 103735:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2921.653995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2921.661038] Lustre: Skipped 1 previous similar message [ 2922.802736] LustreError: 107618:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2922.809536] LustreError: 107618:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2922.819690] LustreError: 103740:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2922.975711] LustreError: 107630:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.125@tcp: cfg command failed: rc = -17 [ 2922.994060] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2922.994060] [ 2927.103615] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2928.900530] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 2944.716865] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 01:44:47 (1787463887) [ 2963.765685] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing load_modules_local [ 2982.544578] Lustre: 109686:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3003.190430] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3006.869915] Lustre: 110821:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3095.520169] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3095.526858] LustreError: Skipped 1 previous similar message [ 3095.530107] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3095.537196] Lustre: Skipped 2 previous similar messages [ 3095.543748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3095.551841] Lustre: Skipped 4 previous similar messages [ 3099.906670] Lustre: server umount lustre-MDT0000 complete [ 3101.157698] LustreError: 104749:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3101.195453] LustreError: 104749:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 3103.559528] LustreError: 106788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787464049 with bad export cookie 6418404675573886538 [ 3103.562444] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3103.565747] LustreError: 106788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3103.917037] Lustre: server umount lustre-MDT0001 complete [ 3118.788322] Lustre: server umount lustre-OST0000 complete [ 3122.399092] Lustre: 93327:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787464052/real 1787464052] req@ffff9afc3ceb0a80 x1874291305317632/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787464068 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3122.453702] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3122.474190] Lustre: Skipped 3 previous similar messages [ 3123.137325] Lustre: server umount lustre-OST0001 complete [ 3129.094033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3139.091208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3146.911069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3157.047575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3175.339834] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing load_modules_local [ 3185.525383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3185.558859] Lustre: lustre-MDT0000: reset Object Index mappings [ 3186.167203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3190.316807] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 3198.986130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3199.015789] Lustre: lustre-MDT0001: reset Object Index mappings [ 3204.006655] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 3206.962562] Lustre: 115922:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3213.899721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3220.922423] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 3223.742243] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 3229.164103] LustreError: 116276:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3229.182992] LustreError: 116276:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3229.605914] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3231.235737] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:382 to 0x280000401:418) [ 3231.349323] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:199 to 0x2c0000400:225) [ 3231.453608] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3236.434339] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 3244.239928] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3248.476345] Lustre: 117798:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3297.814104] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 01:50:41 (1787464241) [ 3327.456261] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3327.463742] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3327.480153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3327.482687] Lustre: Skipped 4 previous similar messages [ 3329.962810] Lustre: server umount lustre-MDT0000 complete [ 3333.739246] LustreError: 114754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787464279 with bad export cookie 6418404675573933634 [ 3333.740499] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3333.752724] LustreError: 114754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3340.292694] Lustre: server umount lustre-MDT0001 complete [ 3350.766626] Lustre: server umount lustre-OST0000 complete [ 3360.707194] Lustre: server umount lustre-OST0001 complete [ 3368.580372] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_hostid [ 3378.775578] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing load_modules_local [ 3422.216700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3422.556808] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3422.624414] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3422.818304] Lustre: lustre-MDT0000: new disk, initializing [ 3423.024082] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3428.587465] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 3438.351739] Lustre: server umount lustre-MDT0000 complete [ 3450.219633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3450.619405] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3450.645936] Lustre: 122071:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3450.668568] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3450.908211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3450.913932] Lustre: Skipped 4 previous similar messages [ 3455.696408] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 3461.300034] Lustre: server umount lustre-MDT0000 complete [ 3471.949121] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 01:53:35 (1787464415) [ 3474.257625] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3476.569076] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 01:53:39 (1787464419) [ 3497.334714] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing load_modules_local [ 3509.644631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3516.198429] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 3526.301690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3526.537887] Lustre: 124127:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3526.658806] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3526.670319] Lustre: Skipped 1 previous similar message [ 3526.833710] Lustre: lustre-MDT0001: new disk, initializing [ 3526.986459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3527.000208] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3532.646609] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 3538.510444] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3546.511119] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3546.865809] Lustre: lustre-OST0000: new disk, initializing [ 3546.869993] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3546.878895] Lustre: 125704:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3548.369459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3548.382754] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3548.496048] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3554.944715] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 3566.786984] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3567.057098] Lustre: lustre-OST0001: new disk, initializing [ 3567.077583] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3567.090734] Lustre: 126576:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3568.475360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3568.495193] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3568.619489] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3575.332323] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 3588.597968] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3593.601580] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4071.743708] Lustre: 133018:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4071.749426] Lustre: 133018:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4071.753484] Lustre: 133018:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4071.756453] Lustre: 133018:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4071.760868] Lustre: 133018:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4071.765257] Lustre: 133018:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4423.111701] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 02:09:25 (1787465365) [ 4439.020185] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4439.022990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4439.044274] Lustre: Skipped 8 previous similar messages [ 4439.049822] Lustre: Skipped 8 previous similar messages [ 4444.136574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4444.143319] Lustre: Skipped 2 previous similar messages [ 4445.353558] Lustre: server umount lustre-MDT0000 complete [ 4449.251895] LustreError: 127413:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4449.288088] LustreError: 127413:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 4449.624352] LustreError: 133076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787465395 with bad export cookie 6418404675574001310 [ 4449.629987] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4449.639945] LustreError: 133076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4454.374399] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4454.376657] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4454.388372] Lustre: Skipped 1 previous similar message [ 4454.419511] Lustre: Skipped 1 previous similar message [ 4456.074799] Lustre: server umount lustre-MDT0001 complete [ 4466.699098] Lustre: server umount lustre-OST0000 complete [ 4477.561117] Lustre: server umount lustre-OST0001 complete [ 4486.637352] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_hostid [ 4498.753207] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing load_modules_local [ 4541.753083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4542.008171] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4542.035692] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4542.110859] Lustre: lustre-MDT0000: new disk, initializing [ 4542.219251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4542.227971] Lustre: Skipped 4 previous similar messages [ 4542.244135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4547.115514] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 4559.914767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4560.061717] Lustre: 136086:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4560.161538] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4560.174832] Lustre: Skipped 1 previous similar message [ 4560.246817] Lustre: lustre-MDT0001: new disk, initializing [ 4560.413280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4560.431531] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4565.541420] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 4580.335126] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4588.114546] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4595.152616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4595.531195] Lustre: lustre-OST0000: new disk, initializing [ 4595.533975] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4595.539514] Lustre: 138055:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4597.063057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4597.080550] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4597.193251] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4606.532791] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 4620.657188] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4635.219530] Lustre: lustre-MDD0000: changelog on [ 4638.511484] Lustre: lustre-MDD0001: changelog on [ 6784.622893] Lustre: DEBUG MARKER: conf-sanity test_135: @@@@@@ FAIL: creation thread is running. Is changelog reader stuck? [ 6797.148423] Lustre: lustre-MDD0001: changelog off [ 6801.103829] Lustre: lustre-MDD0000: changelog off [ 6847.968927] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6847.976407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6847.981596] Lustre: Skipped 1 previous similar message [ 6853.090555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6853.123603] Lustre: Skipped 1 previous similar message [ 6854.237263] Lustre: server umount lustre-OST0000 complete [ 6863.328302] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6863.350030] Lustre: Skipped 1 previous similar message [ 6863.355851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6866.855092] Lustre: server umount lustre-MDT0000 complete [ 6868.448057] LustreError: 136096:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6868.469102] LustreError: 136096:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6871.414875] LustreError: 136881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787467817 with bad export cookie 6418404675574829886 [ 6871.418492] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6871.421902] LustreError: 136881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6871.662387] Lustre: server umount lustre-MDT0001 complete [ 6885.013156] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing unload_modules_local [ 6888.643467] Key type lgssc unregistered [ 6889.272781] LNet: 146861:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6889.284908] LNetError: 146861:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6889.302337] LNet: Removed LNI 192.168.206.125@tcp [ 6889.958167] Key type .llcrypt unregistered [ 6889.960944] Key type ._llcrypt unregistered [ 6919.822276] Key type ._llcrypt registered [ 6919.827819] Key type .llcrypt registered [ 6919.967899] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_hostid [ 6939.684463] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing load_modules_local [ 6941.061413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6941.119844] alg: No test for adler32 (adler32-zlib) [ 6942.212646] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 6942.518085] LNet: Added LNI 192.168.206.125@tcp [8/256/0/180] [ 6944.183262] Key type lgssc registered [ 6945.453081] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6984.263537] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6984.279598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6985.435803] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6985.454263] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6985.492240] Lustre: lustre-MDT0000: new disk, initializing [ 6985.526218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6985.535406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6990.509573] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 7008.424142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7008.613955] Lustre: 150064:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7008.709373] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7008.717274] Lustre: Skipped 1 previous similar message [ 7008.844441] Lustre: lustre-MDT0001: new disk, initializing [ 7009.005915] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7009.069442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7009.083200] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7015.122298] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 7026.523633] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7032.585724] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7041.926283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7042.567364] Lustre: lustre-OST0000: new disk, initializing [ 7042.571864] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7042.601230] Lustre: 152037:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7042.852927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7049.868429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7049.889379] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7050.227159] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7052.848544] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 7064.178752] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7070.967840] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing load_module obdecho/obdecho [ 7073.099323] LustreError: 153040:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 7073.109776] LustreError: 153040:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 7075.216875] LustreError: 153136:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 7091.171851] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7091.176772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7091.197845] Lustre: Skipped 1 previous similar message [ 7091.224376] Lustre: Skipped 2 previous similar messages [ 7094.058876] Lustre: server umount lustre-MDT0000 complete [ 7098.007875] LustreError: 150862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787468043 with bad export cookie 1891307313994801039 [ 7098.010170] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7098.017262] LustreError: 150862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7098.388406] Lustre: server umount lustre-MDT0001 complete [ 7113.020886] Lustre: server umount lustre-OST0000 complete [ 7134.377896] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_hostid [ 7145.665294] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing load_modules_local [ 7186.655315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7187.013404] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7187.043419] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7187.135236] Lustre: lustre-MDT0000: new disk, initializing [ 7187.219678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7187.259516] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7191.909526] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 7205.133280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7205.384189] Lustre: 156853:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7205.506141] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7205.513971] Lustre: Skipped 1 previous similar message [ 7205.663343] Lustre: lustre-MDT0001: new disk, initializing [ 7205.822154] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7205.896290] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7205.915339] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7210.923141] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 7223.750450] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7231.809166] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7241.134352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7241.795196] Lustre: lustre-OST0000: new disk, initializing [ 7241.806605] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7241.822290] Lustre: 158827:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7242.005830] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7247.507040] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7247.518651] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7247.734656] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7249.568252] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 7259.795562] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7264.643449] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7264.888898] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7269.921148] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7270.148762] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7273.456280] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7273.471517] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7273.485164] Lustre: Skipped 1 previous similar message [ 7273.494872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7273.500517] Lustre: Skipped 1 previous similar message [ 7278.573346] LustreError: 158808:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7278.580634] LustreError: 158808:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7278.674431] Lustre: server umount lustre-OST0000 complete [ 7283.681188] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7283.698932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7283.707051] Lustre: Skipped 2 previous similar messages [ 7288.801218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7288.821215] Lustre: Skipped 2 previous similar messages [ 7289.121771] Lustre: server umount lustre-MDT0000 complete [ 7293.815940] LustreError: 158826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787468239 with bad export cookie 1891307313994801998 [ 7293.823162] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7293.840690] LustreError: 158826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7294.645845] Lustre: server umount lustre-MDT0001 complete [ 7299.238286] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 02:57:22 (1787468242) [ 7301.186127] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7303.480077] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 02:57:26 (1787468246) [ 7312.744110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7313.292075] LustreError: 161242:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7313.377480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7318.496637] LustreError: 161241:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7319.121783] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 7323.620864] LustreError: 161242:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7328.742246] LustreError: 161241:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7329.232811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7335.479446] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 7345.077286] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7352.336646] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7359.349909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7359.781759] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7359.790948] Lustre: Skipped 1 previous similar message [ 7365.834938] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 7373.964755] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7393.767902] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7393.785665] Lustre: Skipped 2 previous similar messages [ 7393.788459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7393.807522] Lustre: Skipped 2 previous similar messages [ 7397.234708] Lustre: server umount lustre-MDT0000 complete [ 7398.884777] LustreError: 162189:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7398.894364] LustreError: 162189:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7400.844468] LustreError: 161222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787468346 with bad export cookie 1891307313994802880 [ 7400.847333] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7400.859026] LustreError: 161222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7404.000653] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7404.008822] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7404.015435] Lustre: Skipped 1 previous similar message [ 7415.263235] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7415.578487] Lustre: server umount lustre-MDT0001 complete [ 7421.907836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7433.802528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7433.996907] LustreError: 165107:0:(llog.c:1655:llog_backup()) MGC192.168.206.125@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7434.014576] Lustre: 165107:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.125@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7436.041045] LustreError: 165112:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7436.067119] LustreError: 165112:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 7470.564387] LustreError: 165114:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7470.598515] LustreError: 165114:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 26 previous similar messages [ 7494.623308] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7494.626279] Lustre: 165107:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.125@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7535.611518] LustreError: 165113:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7535.622042] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7535.633250] LustreError: 165113:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 43 previous similar messages [ 7540.101768] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 7540.706675] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7549.535887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7562.222369] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7562.232500] LustreError: Skipped 1 previous similar message [ 7571.427529] LustreError: 148287:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000db2ba7ad x1874295928270208/t0(0) o250->MGC192.168.206.125@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7576.576838] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 7577.082237] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7583.560712] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7585.062781] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7590.294900] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7591.943140] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7618.032149] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7618.044673] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7618.051592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7618.056078] Lustre: Skipped 1 previous similar message [ 7618.075082] Lustre: Skipped 6 previous similar messages [ 7620.772321] Lustre: server umount lustre-MDT0000 complete [ 7625.862260] LustreError: 161221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787468571 with bad export cookie 1891307313994809369 [ 7625.870250] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7625.890318] LustreError: 161221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7626.802611] Lustre: server umount lustre-MDT0001 complete [ 7642.314593] Lustre: server umount lustre-OST0000 complete [ 7665.398837] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_hostid [ 7676.930490] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing load_modules_local [ 7725.843337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7726.069856] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7726.094553] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7726.182245] Lustre: lustre-MDT0000: new disk, initializing [ 7726.264793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7726.270127] Lustre: Skipped 1 previous similar message [ 7726.294814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7731.394270] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 7742.282174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7742.506182] Lustre: 170229:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7742.777444] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7747.323322] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 7756.026225] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7761.455457] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7767.047982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7767.273803] Lustre: lustre-OST0000: new disk, initializing [ 7767.276409] Lustre: Skipped 1 previous similar message [ 7767.279340] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7767.284232] Lustre: Skipped 2 previous similar messages [ 7767.290586] Lustre: 172199:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7772.931344] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 7774.759740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7774.782049] Lustre: Skipped 1 previous similar message [ 7774.790049] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7774.860360] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7782.532205] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7787.480559] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7787.800624] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7792.995871] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7793.271651] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7795.680520] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7795.687777] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7795.687907] LustreError: Skipped 1 previous similar message [ 7795.692881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7795.697125] Lustre: Skipped 2 previous similar messages [ 7801.924819] Lustre: server umount lustre-OST0000 complete [ 7809.504505] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7811.866531] Lustre: server umount lustre-MDT0000 complete [ 7815.538172] LustreError: 170221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787468761 with bad export cookie 1891307313994826169 [ 7815.544905] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7815.552437] LustreError: 170221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7816.095822] Lustre: server umount lustre-MDT0001 complete [ 7828.674751] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 03:06:11 (1787468771) [ 7837.149405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7837.769910] LustreError: 174609:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7837.789563] LustreError: 174609:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 34 previous similar messages [ 7843.853853] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 7851.827788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7857.327241] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 7866.554565] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7876.567362] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7885.689942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7893.036379] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 7901.656552] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7922.461389] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 03:07:45 (1787468865) [ 7928.297884] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7928.301736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7928.301860] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7928.307184] Lustre: Skipped 3 previous similar messages [ 7928.343097] Lustre: Skipped 7 previous similar messages [ 7932.461704] Lustre: server umount lustre-OST0000 complete [ 7950.303593] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7950.499174] Lustre: server umount lustre-MDT0000 complete [ 7954.020522] LustreError: 174592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787468899 with bad export cookie 1891307313994827051 [ 7954.026723] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7954.036267] LustreError: 174592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7954.295689] Lustre: server umount lustre-MDT0001 complete [ 7966.114891] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing unload_modules_local [ 7969.096437] Key type lgssc unregistered [ 7969.416686] LNet: 178371:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7969.425283] LNetError: 178371:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7970.469623] LNet: Removed LNI 192.168.206.125@tcp [ 7971.294638] Key type .llcrypt unregistered [ 7971.301634] Key type ._llcrypt unregistered [ 7994.473861] Key type ._llcrypt registered [ 7994.475500] Key type .llcrypt registered [ 7994.549675] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing load_modules_local [ 7996.250236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7996.403712] alg: No test for adler32 (adler32-zlib) [ 7997.542075] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 7997.851033] LNet: Added LNI 192.168.206.125@tcp [8/256/0/180] [ 7999.634291] Key type lgssc registered [ 8001.839691] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8012.546920] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8012.568358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8028.127471] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8033.247356] LustreError: 179616:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.125@tcp: failed processing log, type 4: rc = -110 [ 8038.367838] LustreError: 179616:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8038.400048] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 8038.415909] LustreError: MGC192.168.206.125@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 8038.445222] LustreError: 179616:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 8038.451941] LustreError: 179616:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 8038.459803] LustreError: 179616:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 8038.551078] Lustre: server umount lustre-OST0000 complete [ 8038.553146] LustreError: 179616:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8047.583251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8049.063697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8053.679383] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 8058.347191] LustreError: 180116:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8058.364644] LustreError: 180116:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8062.142628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8062.658354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8067.321433] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 8075.414650] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8081.466943] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8088.489200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8088.898852] LustreError: 181933:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8088.908174] Lustre: 181933:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.206.125@tcp: invalid local config log lustre-OST0000: rc = -22 [ 8089.115244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8095.476292] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 8103.316230] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8107.490032] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8107.499493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8107.528250] Lustre: Skipped 1 previous similar message [ 8107.556536] Lustre: Skipped 2 previous similar messages [ 8112.608553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8112.620372] Lustre: Skipped 2 previous similar messages [ 8113.049893] Lustre: server umount lustre-MDT0000 complete [ 8117.740947] LustreError: 181357:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8117.747816] LustreError: 181357:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8118.485797] LustreError: 180096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787469064 with bad export cookie 8807544313470847911 [ 8118.489271] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8118.497474] LustreError: 180096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8119.904387] LustreError: 180845:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000009fa2449e x1874297034405248/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8120.115298] Lustre: server umount lustre-MDT0001 complete [ 8134.864823] Lustre: server umount lustre-OST0000 complete [ 8147.605812] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_hostid [ 8157.972244] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing load_modules_local [ 8204.116670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8204.551880] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8204.594110] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8204.705809] Lustre: lustre-MDT0000: new disk, initializing [ 8204.785367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8204.802743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8210.196204] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 8222.407138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8222.532450] Lustre: 185463:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8222.581155] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8222.593861] Lustre: Skipped 1 previous similar message [ 8222.672547] Lustre: lustre-MDT0001: new disk, initializing [ 8222.841719] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8222.917381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8222.946993] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8228.179993] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 8239.421230] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8246.518715] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8253.627948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8253.851931] Lustre: lustre-OST0000: new disk, initializing [ 8253.855381] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8253.860370] Lustre: 187433:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8253.927291] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8261.871852] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 8263.760493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8263.782917] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8263.973667] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8272.715751] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8278.910658] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8279.210917] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8284.419448] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8284.645988] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8289.766060] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8289.769357] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8289.775036] Lustre: Skipped 1 previous similar message [ 8289.778254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8292.838770] Lustre: server umount lustre-OST0000 complete [ 8299.489179] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8299.507026] LustreError: Skipped 1 previous similar message [ 8299.530677] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8299.545532] Lustre: Skipped 1 previous similar message [ 8299.550052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8299.560711] Lustre: Skipped 1 previous similar message [ 8305.122540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8305.135619] Lustre: Skipped 2 previous similar messages [ 8312.291799] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8312.751148] Lustre: server umount lustre-MDT0000 complete [ 8314.340133] LustreError: 185470:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8314.368273] LustreError: 185470:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8318.077753] LustreError: 186261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787469263 with bad export cookie 8807544313470848632 [ 8318.079748] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8318.089335] LustreError: 186261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8318.755490] Lustre: server umount lustre-MDT0001 complete [ 8328.422219] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 03:14:31 (1787469271) [ 8349.670167] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing unload_modules_local [ 8353.445377] Key type lgssc unregistered [ 8353.849172] LNet: 190385:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8353.858558] LNetError: 190385:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8354.919335] LNet: Removed LNI 192.168.206.125@tcp [ 8355.989371] Key type .llcrypt unregistered [ 8355.992903] Key type ._llcrypt unregistered [ 8381.782265] Key type ._llcrypt registered [ 8381.783612] Key type .llcrypt registered [ 8381.849845] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing load_modules_local [ 8382.531385] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8382.617272] alg: No test for adler32 (adler32-zlib) [ 8384.083294] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 8384.493499] LNet: Added LNI 192.168.206.125@tcp [8/256/0/180] [ 8386.439361] Key type lgssc registered [ 8388.131619] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8396.871183] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8396.890510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8398.299886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8404.640606] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 8407.533730] LustreError: 191608:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8407.561470] LustreError: 191608:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8412.649503] LustreError: 191607:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8417.308928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8417.945711] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8423.385209] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 8431.075437] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8438.208469] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8446.289695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8446.580706] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8446.587710] LustreError: 193473:0:(llog.c:1696:llog_backup()) MGC192.168.206.125@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8446.595424] Lustre: 193473:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.206.125@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8446.773838] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8453.667702] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 8460.950154] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8463.843102] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8463.855833] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8463.871897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8464.929943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8464.930124] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8464.935257] Lustre: Skipped 1 previous similar message [ 8464.959808] Lustre: Skipped 1 previous similar message [ 8469.298981] Lustre: server umount lustre-MDT0000 complete [ 8469.985769] LustreError: 191604:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8470.012435] LustreError: 191604:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8474.585069] LustreError: 191588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787469420 with bad export cookie 1476384409040748211 [ 8474.593616] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8474.603823] LustreError: 191588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8475.106395] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8475.109271] LustreError: 191603:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8475.158687] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8475.175404] Lustre: Skipped 1 previous similar message [ 8475.202260] LustreError: 192338:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000ebb6277a x1874297440201472/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8475.528441] Lustre: server umount lustre-MDT0001 complete [ 8481.050436] Lustre: server umount lustre-OST0000 complete [ 8491.824982] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_hostid [ 8503.358685] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing load_modules_local [ 8542.879627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8543.194550] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8543.244413] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8543.363575] Lustre: lustre-MDT0000: new disk, initializing [ 8543.449110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8543.470696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8547.766197] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 8558.398554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8558.537193] Lustre: 196998:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8558.622548] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8558.630054] Lustre: Skipped 1 previous similar message [ 8558.637775] LustreError: 197779:0:(llog.c:1696:llog_backup()) MGC192.168.206.125@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8558.652710] LustreError: 197779:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8558.658213] Lustre: 197779:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.125@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8558.694129] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8558.701124] Lustre: Skipped 1 previous similar message [ 8558.789690] Lustre: lustre-MDT0001: new disk, initializing [ 8558.937194] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8558.979765] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8558.997148] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8563.930832] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 8573.636965] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8579.433870] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8586.338532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8586.563403] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8586.565940] Lustre: Skipped 1 previous similar message [ 8586.569039] LustreError: 198944:0:(llog.c:1696:llog_backup()) MGC192.168.206.125@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8586.574227] LustreError: 198944:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8586.577899] Lustre: 198944:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.125@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8586.583021] Lustre: 198944:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8586.629679] Lustre: lustre-OST0000: new disk, initializing [ 8586.642599] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8586.655251] Lustre: 198967:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8586.734574] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8593.208784] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 8595.006349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8595.018085] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8595.182357] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8603.593700] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8609.668984] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8610.021196] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8615.393685] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8615.656573] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8621.028781] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8621.054487] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8621.057966] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8621.064984] Lustre: Skipped 1 previous similar message [ 8626.157541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8626.165936] Lustre: Skipped 1 previous similar message [ 8632.287426] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8632.452207] Lustre: server umount lustre-OST0000 complete [ 8640.996252] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8641.012775] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8641.037586] Lustre: Skipped 1 previous similar message [ 8641.048421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8641.060407] Lustre: Skipped 2 previous similar messages [ 8643.606967] Lustre: server umount lustre-MDT0000 complete [ 8646.625527] LustreError: 197007:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8646.640739] LustreError: 197007:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8647.683203] LustreError: 196990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787469593 with bad export cookie 1476384409040748925 [ 8647.705036] LustreError: 196990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8647.705853] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8648.074780] Lustre: server umount lustre-MDT0001 complete [ 8657.695686] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 03:20:00 (1787469600) [ 8678.190293] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing load_modules_local [ 8690.631671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8690.997229] LustreError: 202098:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8691.042770] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8696.026278] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 8705.867576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8712.296986] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 8716.158675] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8724.188736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8724.589445] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8724.596651] Lustre: Skipped 1 previous similar message [ 8732.188595] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 8741.221185] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8741.428322] Lustre: lustre-OST0001: new disk, initializing [ 8741.434816] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8741.444899] Lustre: 204407:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8748.170386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8748.189774] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8748.309915] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8749.591878] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 8760.867505] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8773.785261] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8776.491392] Lustre: DEBUG MARKER: ADD OST3 [ 8783.813929] loop: module loaded [ 8783.944095] loop0: detected capacity change from 0 to 204800000 [ 8784.165044] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8784.198328] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8784.268440] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8793.252551] Lustre: DEBUG MARKER: STOP OST3 [ 8793.913766] loop0: detected capacity change from 0 to 204800000 [ 8803.403350] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8803.877195] Lustre: lustre-OST0003: new disk, initializing [ 8803.889400] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8803.915873] Lustre: 206483:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8804.207596] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8804.226119] Lustre: Skipped 1 previous similar message [ 8805.195153] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8805.373722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8805.389208] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8805.456543] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8806.808647] Lustre: Failing over lustre-OST0003 [ 8807.117359] Lustre: server umount lustre-OST0003 complete [ 8807.268639] LustreError: 206541:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8810.359264] LustreError: 203587:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.206.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8810.383563] LustreError: 203587:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 8810.466989] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8810.481854] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8810.501032] Lustre: Skipped 2 previous similar messages [ 8813.347947] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 8825.240530] Lustre: DEBUG MARKER: START OST3 again [ 8833.593391] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8833.906478] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8835.300906] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8835.723112] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8835.723811] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8841.314905] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 8865.169775] Lustre: Failing over lustre-OST0003 [ 8865.349328] Lustre: server umount lustre-OST0003 complete [ 8866.787928] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8866.792393] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8866.797984] LustreError: Skipped 1 previous similar message [ 8866.809911] LustreError: 203587:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8866.817082] Lustre: Skipped 1 previous similar message [ 8866.853443] LustreError: 203587:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 8885.747073] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8885.764704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8885.772514] Lustre: Skipped 2 previous similar messages [ 8891.726237] Lustre: server umount lustre-MDT0000 complete [ 8896.250545] LustreError: 202078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787469841 with bad export cookie 1476384409040749807 [ 8896.255774] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8896.261086] LustreError: 202078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8900.602534] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8900.628777] Lustre: Skipped 5 previous similar messages [ 8902.898515] Lustre: server umount lustre-MDT0001 complete [ 8907.849319] Lustre: server umount lustre-OST0000 complete [ 8912.127605] Lustre: server umount lustre-OST0001 complete [ 8920.636545] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_hostid [ 8931.213959] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing load_modules_local [ 8975.832622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8976.431507] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8976.531351] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8976.665563] Lustre: lustre-MDT0000: new disk, initializing [ 8976.789381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8976.793848] Lustre: Skipped 1 previous similar message [ 8976.820438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8982.481971] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 8995.295739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8995.518748] Lustre: 212024:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8995.599825] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8995.614259] Lustre: Skipped 1 previous similar message [ 8995.743233] Lustre: lustre-MDT0001: new disk, initializing [ 8995.887274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8995.905084] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9002.068860] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 9013.116984] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9020.078430] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9027.916031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9028.301920] Lustre: lustre-OST0000: new disk, initializing [ 9028.310301] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9028.319466] Lustre: 213995:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9030.045066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9030.066596] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9030.173289] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9035.990105] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 9047.778629] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9053.785246] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9054.012717] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9059.577161] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9059.959891] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9065.952700] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9065.953885] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9065.967993] LustreError: Skipped 1 previous similar message [ 9065.968806] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9065.968812] Lustre: Skipped 8 previous similar messages [ 9066.021218] Lustre: Skipped 1 previous similar message [ 9076.195186] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9076.389176] Lustre: server umount lustre-OST0000 complete [ 9086.826237] Lustre: server umount lustre-MDT0000 complete [ 9089.506218] LustreError: 212873:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9089.533082] LustreError: 212873:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 19 previous similar messages [ 9091.446198] LustreError: 212015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787470037 with bad export cookie 1476384409040752103 [ 9091.446924] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9091.458108] LustreError: 212015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9092.074724] Lustre: server umount lustre-MDT0001 complete [ 9096.403318] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 03:27:19 (1787470039) [ 9115.445867] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_hostid [ 9127.585886] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing load_modules_local [ 9169.610457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9169.905260] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9169.959983] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9170.048265] Lustre: lustre-MDT0000: new disk, initializing [ 9170.200356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9175.453582] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 9187.099875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9187.244501] Lustre: 218610:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9187.504417] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9192.569980] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 9203.506186] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9210.231878] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9217.463193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9217.808734] Lustre: 220579:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9219.566331] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9225.250880] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 9234.837938] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9239.637156] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9239.905809] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9245.295331] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9245.529842] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9250.275662] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9250.275883] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9250.281314] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9250.281324] Lustre: Skipped 8 previous similar messages [ 9250.283081] LustreError: Skipped 2 previous similar messages [ 9250.300506] Lustre: Skipped 3 previous similar messages [ 9253.815840] Lustre: server umount lustre-OST0000 complete [ 9263.923557] Lustre: server umount lustre-MDT0000 complete [ 9265.637168] LustreError: 218620:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9265.667958] LustreError: 218620:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9267.499376] LustreError: 219404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787470213 with bad export cookie 1476384409040752985 [ 9267.510997] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9267.522047] LustreError: 219404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9268.009072] Lustre: server umount lustre-MDT0001 complete [ 9276.788868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9277.516779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9277.520922] Lustre: Skipped 5 previous similar messages [ 9282.371293] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 9291.099861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9295.675626] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 9303.451350] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9309.524272] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9316.728110] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9323.621409] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 9331.716905] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9342.956531] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9342.966495] LustreError: Skipped 1 previous similar message [ 9348.066362] Lustre: server umount lustre-OST0000 complete [ 9357.880996] Lustre: server umount lustre-MDT0000 complete [ 9361.622538] LustreError: 222776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787470307 with bad export cookie 1476384409040753867 [ 9361.624657] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9361.634127] LustreError: 222776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9362.169764] Lustre: server umount lustre-MDT0001 complete [ 9473.562974] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing unload_modules_local [ 9476.728646] Key type lgssc unregistered [ 9477.154623] LNet: 226468:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9477.166696] LNetError: 226468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9477.191902] LNet: Removed LNI 192.168.206.125@tcp [ 9478.271201] Key type .llcrypt unregistered [ 9478.274235] Key type ._llcrypt unregistered [ 9506.285089] Key type ._llcrypt registered [ 9506.288201] Key type .llcrypt registered [ 9506.351242] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_hostid [ 9523.850473] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing load_modules_local [ 9524.568628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9524.619083] alg: No test for adler32 (adler32-zlib) [ 9525.595382] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [ 9525.864472] LNet: Added LNI 192.168.206.125@tcp [8/256/0/180] [ 9527.535689] Key type lgssc registered [ 9528.568089] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9567.325174] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9567.355503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9568.562510] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9568.586695] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9568.658776] Lustre: lustre-MDT0000: new disk, initializing [ 9568.742673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9568.766122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9572.855205] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 9584.708274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9584.902990] Lustre: 229822:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9585.029574] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9585.034056] Lustre: Skipped 1 previous similar message [ 9585.204113] Lustre: lustre-MDT0001: new disk, initializing [ 9585.305538] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9585.336165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9585.346086] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9591.125962] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 9601.012986] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9607.508571] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9614.260864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9614.570564] Lustre: lustre-OST0000: new disk, initializing [ 9614.576288] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9614.584552] Lustre: 231790:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9614.665307] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9621.333604] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 9623.135855] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9623.142591] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9623.253632] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9630.046271] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9634.868252] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9635.093710] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9639.480685] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9639.631924] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9643.999883] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9644.003551] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9644.005878] LustreError: Skipped 1 previous similar message [ 9644.020231] Lustre: Skipped 1 previous similar message [ 9644.023030] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9644.039887] Lustre: Skipped 1 previous similar message [ 9647.530180] Lustre: server umount lustre-OST0000 complete [ 9652.197695] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9652.207246] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9652.232834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9653.233200] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9653.259630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9653.271288] Lustre: Skipped 1 previous similar message [ 9657.458905] Lustre: server umount lustre-MDT0000 complete [ 9661.607255] LustreError: 229813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787470607 with bad export cookie 16570852674918351377 [ 9661.614096] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9661.616606] LustreError: 229813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9662.126817] Lustre: server umount lustre-MDT0001 complete [ 9680.494151] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing load_modules_local [ 9690.540734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9690.981911] LustreError: 234725:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9691.003318] LustreError: 234725:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9691.037167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9695.721939] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 9696.243810] LustreError: 234726:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9696.264887] LustreError: 234726:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9700.332496] LustreError: 234725:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9704.746500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9705.197158] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9710.198970] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 9713.206720] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9719.885744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9720.222567] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9727.668600] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 9736.572046] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9736.711465] Lustre: lustre-OST0001: new disk, initializing [ 9736.714657] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9736.719232] Lustre: 237033:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9736.818071] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9743.090449] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 9745.075694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9745.094755] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9745.225534] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9755.051622] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9762.687893] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9774.538457] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 03:38:37 (1787470717) [ 9777.120749] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9777.131082] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9777.142601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9781.220051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9781.238246] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9781.253870] Lustre: Skipped 1 previous similar message [ 9783.159564] Lustre: server umount lustre-MDT0000 complete [ 9786.346792] LustreError: 234721:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9786.370576] LustreError: 234721:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9786.800673] LustreError: 237800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787470732 with bad export cookie 16570852674918352259 [ 9786.802101] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9786.817037] LustreError: 237800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9787.115897] Lustre: server umount lustre-MDT0001 complete [ 9800.789671] Lustre: server umount lustre-OST0000 complete [ 9815.405779] Lustre: server umount lustre-OST0001 complete [ 9824.064839] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_hostid [ 9835.069607] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing load_modules_local [ 9874.028907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9874.304604] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9874.339725] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9874.423351] Lustre: lustre-MDT0000: new disk, initializing [ 9874.558400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9874.578877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9879.596882] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 9890.966691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9891.078510] Lustre: 240978:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9891.127317] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9891.130488] Lustre: Skipped 1 previous similar message [ 9891.194372] Lustre: lustre-MDT0001: new disk, initializing [ 9891.316851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9891.333994] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9896.057570] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 9905.854613] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9911.483331] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9918.651468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9918.995480] Lustre: lustre-OST0000: new disk, initializing [ 9919.000282] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9919.018883] Lustre: 242946:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9920.599939] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9920.622310] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9920.736755] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9925.587782] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [ 9934.673309] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9939.764392] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9939.929903] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9945.980058] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9946.340426] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9951.724415] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9951.740651] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9951.764760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9951.770622] Lustre: Skipped 4 previous similar messages [ 9954.836515] Lustre: server umount lustre-OST0000 complete [ 9972.196988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9972.206171] Lustre: Skipped 5 previous similar messages [ 9973.215135] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9973.395912] Lustre: server umount lustre-MDT0000 complete [ 9977.329731] LustreError: 242139:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9977.355671] LustreError: 242139:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9977.787752] LustreError: 242945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787470923 with bad export cookie 16570852674918354394 [ 9977.797270] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9977.800554] LustreError: 242945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9978.084645] Lustre: server umount lustre-MDT0001 complete [ 9987.950988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9988.418047] LustreError: 245165:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9988.492591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9988.500294] Lustre: Skipped 2 previous similar messages [ 9993.657611] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [10003.439773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10009.135367] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [10017.617599] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10024.049567] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10032.258039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10039.614138] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [10048.168401] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10198.001018] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10198.006866] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10198.015298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10198.019047] LustreError: Skipped 1 previous similar message [10198.067793] Lustre: Skipped 4 previous similar messages [10203.313914] Lustre: server umount lustre-OST0000 complete [10221.025620] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10221.195036] Lustre: server umount lustre-MDT0000 complete [10223.589048] LustreError: 245161:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10223.607111] LustreError: 245161:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10224.748511] LustreError: 247013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787471170 with bad export cookie 16570852674918355276 [10224.759287] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10224.764138] LustreError: 247013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10225.063616] Lustre: server umount lustre-MDT0001 complete [10315.493940] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing unload_modules_local [10318.400367] Key type lgssc unregistered [10318.673826] LNet: 248782:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10318.683735] LNetError: 248782:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10318.702544] LNet: Removed LNI 192.168.206.125@tcp [10319.643545] Key type .llcrypt unregistered [10319.648106] Key type ._llcrypt unregistered [10343.873696] Key type ._llcrypt registered [10343.879121] Key type .llcrypt registered [10344.017926] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_hostid [10356.125539] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing load_modules_local [10356.703856] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10356.733838] alg: No test for adler32 (adler32-zlib) [10357.721223] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [10357.887945] LNet: Added LNI 192.168.206.125@tcp [8/256/0/180] [10359.527215] Key type lgssc registered [10360.593562] Lustre: Echo OBD driver; http://www.lustre.org/ [10398.365915] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing load_modules_local [10406.348866] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10406.362907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10407.608839] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10407.628638] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10407.685440] Lustre: lustre-MDT0000: new disk, initializing [10407.772801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10407.785648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10412.044241] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [10421.538518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10421.662872] Lustre: 252829:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10421.786635] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10421.793226] Lustre: Skipped 1 previous similar message [10421.865463] Lustre: lustre-MDT0001: new disk, initializing [10421.942142] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10421.974520] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10421.986521] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10425.606102] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [10429.613712] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10435.708246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10435.944963] Lustre: lustre-OST0000: new disk, initializing [10435.948136] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10435.952048] Lustre: 254464:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10436.050212] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10439.293381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10439.306915] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10439.422520] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10441.778287] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [10453.248534] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10453.410191] Lustre: lustre-OST0001: new disk, initializing [10453.413446] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10453.428164] Lustre: 255335:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10453.591826] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10462.284907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10462.306830] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10462.423061] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10462.512244] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [10475.605845] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10483.598477] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10498.532416] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10498.550222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10498.573700] Lustre: Skipped 3 previous similar messages [10498.578137] Lustre: Skipped 2 previous similar messages [10502.211179] Lustre: server umount lustre-MDT0000 complete [10503.655378] LustreError: 252837:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10503.670547] LustreError: 252837:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10506.134794] LustreError: 252824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787471451 with bad export cookie 2470215320753454115 [10506.141024] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10506.143831] LustreError: 252824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10506.662721] Lustre: server umount lustre-MDT0001 complete [10522.072637] Lustre: server umount lustre-OST0000 complete [10524.897787] Lustre: 250339:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787471454/real 1787471454] req@ffff9afc35263b80 x1874299509093888/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787471470 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10524.922265] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10526.849439] Lustre: server umount lustre-OST0001 complete [10542.716804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10552.679939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10560.372486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10570.674475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10591.554500] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing load_modules_local [10602.757501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10602.796957] Lustre: lustre-MDT0000: reset Object Index mappings [10603.433696] LustreError: 260345:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10603.449539] LustreError: 260345:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10603.515363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10608.561037] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [10608.611850] LustreError: 260346:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10613.741045] LustreError: 260346:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10617.643035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10617.656475] Lustre: lustre-MDT0001: reset Object Index mappings [10618.220022] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10623.002765] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [10626.252229] Lustre: 261486:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10633.174197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10639.325866] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [10642.731473] LustreError: 261839:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10647.564320] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10647.831310] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10647.842332] Lustre: Skipped 1 previous similar message [10648.878367] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10648.900940] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10653.557437] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [10661.013732] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10664.346654] Lustre: 263359:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10669.538024] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10669.543697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10669.559529] Lustre: Skipped 2 previous similar messages [10669.580316] Lustre: Skipped 4 previous similar messages [10670.418690] Lustre: server umount lustre-MDT0000 complete [10674.017239] LustreError: 260325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787471619 with bad export cookie 2470215320753460905 [10674.025861] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10674.029427] LustreError: 260325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10674.470284] Lustre: server umount lustre-MDT0001 complete [10688.243573] Lustre: server umount lustre-OST0000 complete [10691.167215] Lustre: 250339:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787471620/real 1787471620] req@ffff9afc27c1ca80 x1874299509138304/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787471636 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10691.209306] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10691.220241] Lustre: Skipped 1 previous similar message [10693.149106] Lustre: server umount lustre-OST0001 complete [10715.919607] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 03:54:19 (1787471659) [10733.512976] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_hostid [10743.660165] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing load_modules_local [10778.320710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10778.503751] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10778.535054] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10778.617930] Lustre: lustre-MDT0000: new disk, initializing [10778.694405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10778.715595] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10782.330786] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [10791.398960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10791.492601] Lustre: 267616:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10791.567634] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10791.571573] Lustre: Skipped 1 previous similar message [10791.636181] Lustre: lustre-MDT0001: new disk, initializing [10791.801443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10791.821146] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10795.015275] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [10802.971991] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10807.897167] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10813.760280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10814.045592] Lustre: lustre-OST0000: new disk, initializing [10814.053905] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10814.070493] Lustre: 269582:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10815.378764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10815.391711] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10815.522074] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10820.226929] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [10829.804795] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10835.154947] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10835.360773] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10839.150093] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10839.316057] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10841.062324] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10841.062576] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10841.064656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10841.073062] Lustre: Skipped 1 previous similar message [10841.076265] LustreError: Skipped 1 previous similar message [10846.182032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10846.187572] Lustre: Skipped 2 previous similar messages [10851.303248] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10851.313859] Lustre: Skipped 1 previous similar message [10854.883117] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10855.104066] Lustre: server umount lustre-OST0000 complete [10861.536640] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10861.542736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10861.542939] Lustre: Skipped 1 previous similar message [10861.549450] Lustre: Skipped 1 previous similar message [10874.335450] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10874.557608] Lustre: server umount lustre-MDT0000 complete [10876.901902] LustreError: 269964:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10876.920612] LustreError: 269964:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10878.509293] LustreError: 267607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787471824 with bad export cookie 2470215320753463005 [10878.517637] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10878.526522] LustreError: 267607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10878.789514] Lustre: server umount lustre-MDT0001 complete [10899.791958] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing load_modules_local [10910.082811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10910.545577] LustreError: 272520:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10910.592987] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10910.595628] Lustre: Skipped 2 previous similar messages [10915.255477] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [10923.970830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10928.806751] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [10932.002801] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10938.711129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10945.653395] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [10953.835548] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10954.114543] Lustre: lustre-OST0001: new disk, initializing [10954.118425] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10954.124617] Lustre: 274829:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10955.821992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10955.859708] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10955.970885] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10960.843491] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [10970.332172] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10973.941941] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10979.133860] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10979.205837] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10984.823089] Lustre: server umount lustre-MDT0000 complete [10985.953829] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10985.960525] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10985.984065] LustreError: 272521:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10986.007727] LustreError: 272521:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10988.169915] LustreError: 272501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787471933 with bad export cookie 2470215320753463887 [10988.176434] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10988.184929] LustreError: 272501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10990.515747] Lustre: server umount lustre-MDT0001 complete [11004.159431] Lustre: server umount lustre-OST0000 complete [11017.972991] Lustre: server umount lustre-OST0001 complete [11037.842631] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing load_modules_local [11045.615762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11046.047716] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11046.054794] Lustre: Skipped 3 previous similar messages [11050.441931] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [11051.501946] LustreError: 277886:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11051.528570] LustreError: 277886:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [11058.923938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11064.663957] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [11067.732643] Lustre: 279027:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11073.621375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11079.538601] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [11087.599611] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11091.036260] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [11091.096586] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [11094.510109] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [11101.710569] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11104.844346] Lustre: 280904:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11109.077629] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [11109.096176] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [11118.037767] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 04:01:01 (1787472061) [11135.981339] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11135.993343] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11136.007918] Lustre: Skipped 3 previous similar messages [11136.011207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11136.013769] Lustre: Skipped 4 previous similar messages [11139.119835] Lustre: server umount lustre-MDT0000 complete [11142.348552] LustreError: 280206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787472088 with bad export cookie 2470215320753470187 [11142.358423] LustreError: 280206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11142.358659] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11148.764829] Lustre: server umount lustre-MDT0001 complete [11158.669478] Lustre: server umount lustre-OST0000 complete [11168.895882] Lustre: server umount lustre-OST0001 complete [11176.624802] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_hostid [11193.020110] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing load_modules_local [11239.910813] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing load_modules_local [11252.537033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11253.012467] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11253.054439] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11253.210969] Lustre: lustre-MDT0000: new disk, initializing [11253.453542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11258.914527] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [11268.815666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11268.989932] Lustre: 285176:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11269.273873] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11273.822576] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [11285.398357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11285.728848] Lustre: 286811:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11287.755581] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11292.007117] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [11301.729216] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11301.921726] Lustre: 287682:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11303.878771] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11303.892121] Lustre: Skipped 1 previous similar message [11304.009266] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11308.934858] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [11319.376659] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11323.453732] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11323.457730] Lustre: Skipped 1 previous similar message [11339.659360] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 04:04:42 (1787472282) [11344.211071] Lustre: server umount lustre-MDT0000 complete [11344.884436] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11344.888334] LustreError: 287689:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11344.904885] Lustre: Skipped 6 previous similar messages [11344.921616] LustreError: 287689:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [11348.074696] LustreError: 285170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787472293 with bad export cookie 2470215320753478055 [11348.089936] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11348.093560] LustreError: 285170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11348.441623] Lustre: server umount lustre-MDT0001 complete [11361.517575] Lustre: server umount lustre-OST0000 complete [11375.812361] Lustre: server umount lustre-OST0001 complete [11384.494607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11385.096948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11385.101178] Lustre: Skipped 7 previous similar messages [11390.091965] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [11399.197221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11404.267290] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [11411.887818] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11418.251111] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11425.167803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11432.202463] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [11439.653414] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11459.250400] loop1: detected capacity change from 0 to 204800000 [11459.494549] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11459.521293] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11459.552470] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11464.454931] loop1: detected capacity change from 0 to 204800000 [11470.025891] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11470.125717] LustreError: 289919:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs12930: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11470.140161] LustreError: 289919:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs12930 [11470.148519] LustreError: 289919:0:(mgs_handler.c:517:mgs_target_reg()) fs12930-MDT0001: Error checking target index: rc = -13 [11470.158938] LustreError: fs12930-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11470.167474] LustreError: 293650:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11470.172503] LustreError: 293650:0:(tgt_mount.c:2129:server_put_super()) no obd fs12930-MDT0001 [11470.176709] LustreError: 293650:0:(tgt_mount.c:116:server_deregister_mount()) fs12930-MDT0001 not registered [11470.232090] Lustre: server umount fs12930-MDT0001 complete [11470.240178] LustreError: 293650:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11478.960259] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11479.067842] Lustre: Setting parameter fs12930-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs12930-MDT0001 [11479.142457] Lustre: srv-fs12930-MDT0001: No data found on store. Initialize space. [11479.154860] Lustre: Skipped 4 previous similar messages [11479.294921] Lustre: fs12930-MDT0001: new disk, initializing [11479.307090] Lustre: Skipped 3 previous similar messages [11479.415863] Lustre: srv-fs12930-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11479.427489] Lustre: 294116:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs12930-MDT0001-mdtlov: can't create new llog: rc = -115 [11479.447219] LustreError: 294116:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs12930-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11483.753608] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [11490.404575] Lustre: server umount fs12930-MDT0001 complete [11506.653312] loop1: detected capacity change from 0 to 204800000 [11511.782902] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11512.681034] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11513.378114] Lustre: 295686:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs12930-MDT0001-mdtlov: can't create new llog: rc = -115 [11513.382470] LustreError: 295686:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs12930-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11518.924519] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [11524.266180] Lustre: server umount fs12930-MDT0001 complete [11527.172988] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11537.382083] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11537.383810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11537.394809] Lustre: Skipped 4 previous similar messages [11537.398119] Lustre: Skipped 9 previous similar messages [11548.129197] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11548.436170] Lustre: server umount lustre-MDT0000 complete [11558.349899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11558.506619] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11563.290569] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [11564.032043] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11571.083788] loop1: detected capacity change from 0 to 204800000 [11576.537187] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11577.061125] Lustre: 298135:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs12930-MDT0001-mdtlov: can't create new llog: rc = -115 [11577.073165] LustreError: 298135:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs12930-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11582.447722] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [11588.623138] Lustre: server umount fs12930-MDT0001 complete [11604.302616] LustreError: 291761:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11604.317039] LustreError: 291761:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 109 previous similar messages [11614.176494] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11614.213501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11614.221709] Lustre: Skipped 10 previous similar messages [11625.951643] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11626.291435] Lustre: server umount lustre-OST0000 complete [11630.566657] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11630.577175] LustreError: Skipped 1 previous similar message [11644.383367] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11644.578769] Lustre: server umount lustre-MDT0000 complete [11648.325086] LustreError: 289911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787472594 with bad export cookie 2470215320753480736 [11648.339593] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11648.343103] LustreError: 289911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11648.755085] Lustre: server umount lustre-MDT0001 complete [11660.449647] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing unload_modules_local [11663.234692] Key type lgssc unregistered [11663.456859] LNet: 300486:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11663.482975] LNetError: 300486:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11663.508233] LNet: Removed LNI 192.168.206.125@tcp [11664.268158] Key type .llcrypt unregistered [11664.269694] Key type ._llcrypt unregistered [11683.120875] Key type ._llcrypt registered [11683.127278] Key type .llcrypt registered [11683.270354] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing load_modules_local [11684.688667] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11684.751172] alg: No test for adler32 (adler32-zlib) [11685.753478] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [11686.128441] LNet: Added LNI 192.168.206.125@tcp [8/256/0/180] [11687.855259] Key type lgssc registered [11688.760605] Lustre: Echo OBD driver; http://www.lustre.org/ [11696.899310] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11696.915358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11698.408643] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11703.871858] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [11708.903617] LustreError: 301758:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11708.923616] LustreError: 301758:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11709.937781] LustreError: 301759:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11713.283466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11713.836705] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11718.381588] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [11726.061488] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11732.482314] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11740.280951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11740.803875] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11745.906183] LustreError: 303591:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11747.816425] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [11751.407228] LustreError: 303590:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11756.192803] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11756.518430] LustreError: 303589:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11764.710962] LustreError: 303590:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11764.741482] LustreError: 303590:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11776.197368] loop1: detected capacity change from 0 to 204800000 [11776.389136] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11776.399792] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11776.416922] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11782.207637] loop1: detected capacity change from 0 to 204800000 [11784.007588] LustreError: 303589:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11784.047630] LustreError: 303589:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [11788.010563] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11788.221752] Lustre: fs2423-OST0000: new disk, initializing [11788.231444] Lustre: srv-fs2423-OST0000: No data found on store. Initialize space. [11788.244467] Lustre: 305496:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11788.315368] Lustre: fs2423-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11794.445507] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [11802.660826] Lustre: server umount fs2423-OST0000 complete [11814.944713] loop1: detected capacity change from 0 to 204800000 [11819.843269] LustreError: 303859:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11819.873847] LustreError: 303859:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 20 previous similar messages [11820.037988] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11820.276824] Lustre: fs2423-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11825.708895] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [11826.152313] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11834.017224] loop2: detected capacity change from 0 to 204800000 [11834.188532] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11834.203724] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11834.211241] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11839.166591] loop2: detected capacity change from 0 to 204800000 [11841.506887] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11844.731421] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11844.794033] LustreError: fs2423-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11844.806897] LustreError: fs2423-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11844.818888] LustreError: 308535:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11844.825494] LustreError: 308535:0:(tgt_mount.c:2129:server_put_super()) no obd fs2423-OST0001 [11844.832092] LustreError: 308535:0:(tgt_mount.c:116:server_deregister_mount()) fs2423-OST0001 not registered [11844.861444] Lustre: server umount fs2423-OST0001 complete [11844.864657] LustreError: 308535:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11855.157597] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11855.305432] Lustre: fs2423-OST0001: new disk, initializing [11855.311477] Lustre: srv-fs2423-OST0001: No data found on store. Initialize space. [11855.318983] Lustre: 309087:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11855.422664] Lustre: fs2423-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11861.500813] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [11870.084560] Lustre: server umount fs2423-OST0000 complete [11879.559321] Lustre: server umount fs2423-OST0001 complete [11886.406661] LustreError: 309946:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11886.420336] LustreError: 309946:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [11892.087311] loop1: detected capacity change from 0 to 204800000 [11896.964315] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11897.277147] Lustre: fs2423-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11903.732926] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [11911.565910] loop2: detected capacity change from 0 to 204800000 [11916.567879] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11923.750127] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [11931.901684] Lustre: server umount fs2423-OST0000 complete [11941.415397] Lustre: server umount fs2423-OST0001 complete [11954.147732] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11954.150362] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11954.165490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11954.190039] Lustre: Skipped 1 previous similar message [11959.267360] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11959.274723] Lustre: Skipped 2 previous similar messages [11959.598276] Lustre: server umount lustre-OST0000 complete [11964.385699] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11964.401896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11964.408037] Lustre: Skipped 1 previous similar message [11964.426251] Lustre: Skipped 1 previous similar message [11969.579707] Lustre: server umount lustre-MDT0000 complete [11973.555529] LustreError: 314259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787472919 with bad export cookie 10296134016909854982 [11973.558758] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11973.575077] LustreError: 314259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11973.977850] Lustre: server umount lustre-MDT0001 complete [11986.084857] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing unload_modules_local [11988.469556] Key type lgssc unregistered [11988.676406] LNet: 315014:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11988.686351] LNetError: 315014:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11988.707576] LNet: Removed LNI 192.168.206.125@tcp [11989.372134] Key type .llcrypt unregistered [11989.373772] Key type ._llcrypt unregistered [12015.965344] Key type ._llcrypt registered [12015.971939] Key type .llcrypt registered [12016.041625] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_hostid [12032.037789] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing load_modules_local [12033.189525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12033.228446] alg: No test for adler32 (adler32-zlib) [12034.354718] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [12034.601683] LNet: Added LNI 192.168.206.125@tcp [8/256/0/180] [12036.327901] Key type lgssc registered [12037.361235] Lustre: Echo OBD driver; http://www.lustre.org/ [12073.790545] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12073.816322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12075.079967] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12075.122522] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12075.219791] Lustre: lustre-MDT0000: new disk, initializing [12075.302701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12075.332848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12079.456846] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [12090.167185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12090.336824] Lustre: 318874:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12090.418552] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12090.421596] Lustre: Skipped 1 previous similar message [12090.490559] Lustre: lustre-MDT0001: new disk, initializing [12090.602529] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12090.638331] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12090.649460] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12095.122833] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [12105.445605] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12111.920217] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12118.236481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12118.521891] Lustre: lustre-OST0000: new disk, initializing [12118.528667] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12118.544131] Lustre: 320843:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12118.684394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12124.748157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12124.782594] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12124.904046] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12125.300930] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [12134.788130] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12136.704219] Lustre: Mounted lustre-client - version 2.17.57_65_gc740cad [12137.534347] Lustre: Unmounted lustre-client [12145.634752] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12145.635203] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12145.650489] LustreError: Skipped 1 previous similar message [12145.655276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12145.688538] Lustre: Skipped 1 previous similar message [12147.175820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12147.186462] Lustre: Skipped 2 previous similar messages [12151.551341] Lustre: server umount lustre-OST0000 complete [12157.408221] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12157.415115] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12157.426426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12157.443695] Lustre: Skipped 1 previous similar message [12157.466552] Lustre: Skipped 1 previous similar message [12162.528238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12167.654221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12167.657404] Lustre: Skipped 2 previous similar messages [12169.183135] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12169.352434] Lustre: server umount lustre-MDT0000 complete [12172.557724] LustreError: 318869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787473118 with bad export cookie 12010027574421333640 [12172.566283] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12172.567556] LustreError: 318869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12172.875620] Lustre: server umount lustre-MDT0001 complete [12182.185685] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing unload_modules_local [12184.708775] Key type lgssc unregistered [12191.136590] LNet: 323007:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12191.156629] LNetError: 323007:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12269.031034] LNet: Removed LNI 192.168.206.125@tcp [12269.732951] Key type .llcrypt unregistered [12269.736401] Key type ._llcrypt unregistered [12287.231606] Key type ._llcrypt registered [12287.237374] Key type .llcrypt registered [12287.369162] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_hostid [12303.293783] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing load_modules_local [12304.329293] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12304.352052] alg: No test for adler32 (adler32-zlib) [12305.318173] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [12305.615694] LNet: Added LNI 192.168.206.125@tcp [8/256/0/180] [12307.303242] Key type lgssc registered [12308.409711] Lustre: Echo OBD driver; http://www.lustre.org/ [12336.083803] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 04:21:19 (1787473279) [12341.721927] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12341.764406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12343.043411] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12343.078819] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12343.159859] Lustre: lustre-MDT0000: new disk, initializing [12343.237391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12343.250648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12347.154451] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [12356.459790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12356.620233] Lustre: 326241:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12356.681467] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12356.684745] Lustre: Skipped 1 previous similar message [12356.790726] Lustre: lustre-MDT0001: new disk, initializing [12356.903749] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12356.946628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12356.954948] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12360.985525] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [12368.592553] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12373.038844] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12378.333102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12378.535409] Lustre: lustre-OST0000: new disk, initializing [12378.538566] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12378.543751] Lustre: 328210:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12378.608355] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12383.336838] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [12388.374696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12388.384650] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12388.478598] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12391.315201] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12408.802591] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12408.819255] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12408.819820] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12410.668666] Lustre: server umount lustre-OST0000 complete [12418.529572] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12418.538255] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12418.548270] Lustre: Skipped 1 previous similar message [12418.553099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12418.558157] Lustre: Skipped 1 previous similar message [12420.745505] Lustre: server umount lustre-MDT0000 complete [12424.784666] LustreError: 327046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787473370 with bad export cookie 14444819287222910267 [12424.790456] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12424.794957] LustreError: 327046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12425.073260] Lustre: server umount lustre-MDT0001 complete [12437.723049] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing unload_modules_local [12440.698680] Key type lgssc unregistered [12441.041392] LNet: 330118:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12441.051160] LNetError: 330118:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12441.069970] LNet: Removed LNI 192.168.206.125@tcp [12441.839374] Key type .llcrypt unregistered [12441.841894] Key type ._llcrypt unregistered [12460.825621] Key type ._llcrypt registered [12460.829267] Key type .llcrypt registered [12460.933421] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing load_modules_local [12461.831480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12461.917642] alg: No test for adler32 (adler32-zlib) [12462.962156] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [12463.161793] LNet: Added LNI 192.168.206.125@tcp [8/256/0/180] [12464.799462] Key type lgssc registered [12465.758032] Lustre: Echo OBD driver; http://www.lustre.org/ [12472.593250] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12472.616196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12473.997743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12477.889893] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [12483.562981] LustreError: 331395:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12483.579435] LustreError: 331395:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12485.469061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12485.836335] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12490.326677] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [12496.875311] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12501.886962] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12507.669897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12508.072639] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12513.806560] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [12519.887502] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12529.221068] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12530.440596] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12533.735316] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12533.749725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12533.751988] Lustre: Skipped 2 previous similar messages [12533.766975] Lustre: Skipped 2 previous similar messages [12537.710715] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.25@tcp (stopping) [12538.869351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12538.878099] Lustre: Skipped 2 previous similar messages [12539.915530] Lustre: server umount lustre-MDT0000 complete [12542.783525] LustreError: 332122:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12543.394700] LustreError: 331376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787473489 with bad export cookie 9695978151460283076 [12543.398262] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12543.413488] LustreError: 331376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12543.971551] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12543.973293] LustreError: 331391:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12543.979385] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12547.903855] LustreError: 331392:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12549.089475] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12549.095767] Lustre: Skipped 1 previous similar message [12549.843708] Lustre: server umount lustre-MDT0001 complete [12553.809694] Lustre: server umount lustre-OST0000 complete [12562.463247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12566.656405] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [12569.198902] Lustre: 335576:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.125@tcp [12571.112572] Lustre: server umount MGS complete [12579.351543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12579.808409] LustreError: 336147:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12579.829099] LustreError: 336147:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12579.881535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12584.384914] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [12587.843127] LustreError: 336143:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12587.859691] LustreError: 336143:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12592.466452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12592.825254] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12596.907450] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [12604.050456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12604.367455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12610.076389] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [12617.614976] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12618.939868] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12623.970695] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12625.238972] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12631.009527] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12631.017417] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12631.022921] LustreError: Skipped 1 previous similar message [12631.044278] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12631.269385] Lustre: server umount lustre-OST0000 complete [12638.176539] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12638.197283] Lustre: Skipped 1 previous similar message [12641.056822] Lustre: server umount lustre-MDT0000 complete [12643.305055] LustreError: 336147:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12643.338994] LustreError: 336147:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12644.536570] LustreError: 336128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787473590 with bad export cookie 9695978151460283916 [12644.546384] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12644.547308] LustreError: 336128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12655.196815] Lustre: server umount lustre-MDT0001 complete [12665.214846] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing unload_modules_local [12667.644869] Key type lgssc unregistered [12667.916635] LNet: 339590:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12667.931993] LNetError: 339590:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12668.965807] LNet: Removed LNI 192.168.206.125@tcp [12669.853277] Key type .llcrypt unregistered [12669.857604] Key type ._llcrypt unregistered [12688.701599] Key type ._llcrypt registered [12688.705356] Key type .llcrypt registered [12688.857056] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_hostid [12703.322395] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing load_modules_local [12703.953328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12704.074699] alg: No test for adler32 (adler32-zlib) [12705.155740] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [12705.410357] LNet: Added LNI 192.168.206.125@tcp [8/256/0/180] [12707.103170] Key type lgssc registered [12708.278826] Lustre: Echo OBD driver; http://www.lustre.org/ [12746.161751] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12746.191654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12747.656951] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12747.690946] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12747.825554] Lustre: lustre-MDT0000: new disk, initializing [12747.952720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12747.967359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12753.066116] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [12764.349239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12764.515903] Lustre: 342754:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12764.588947] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12764.592552] Lustre: Skipped 1 previous similar message [12764.704217] Lustre: lustre-MDT0001: new disk, initializing [12764.896608] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12764.941838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12764.957310] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12769.390986] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [12779.959217] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12786.261267] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12795.073721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12795.326591] Lustre: lustre-OST0000: new disk, initializing [12795.332270] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12795.337996] Lustre: 344724:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12795.464389] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12802.685863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12802.711371] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12802.903197] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12803.168154] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [12813.866028] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12819.527910] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12819.827669] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12825.910521] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12826.118211] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12828.644052] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12828.645015] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12828.662404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12828.669699] LustreError: Skipped 1 previous similar message [12828.681438] Lustre: Skipped 1 previous similar message [12833.766780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12833.777917] Lustre: Skipped 2 previous similar messages [12834.453798] Lustre: server umount lustre-OST0000 complete [12838.883843] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12838.907681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12838.911293] Lustre: Skipped 1 previous similar message [12844.002136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12844.014552] Lustre: Skipped 1 previous similar message [12845.038445] Lustre: server umount lustre-MDT0000 complete [12849.326183] LustreError: 344723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787473795 with bad export cookie 14672980647491150340 [12849.329217] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12849.340875] LustreError: 344723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12849.706630] Lustre: server umount lustre-MDT0001 complete [12858.428930] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 04:30:01 (1787473801) [12874.343767] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_hostid [12884.848880] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing load_modules_local [12923.603105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12923.939563] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12923.976838] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12924.089557] Lustre: lustre-MDT0000: new disk, initializing [12924.199108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12924.224567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12928.730933] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [12940.476146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12940.617837] Lustre: 349481:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12940.680944] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12940.687031] Lustre: Skipped 1 previous similar message [12940.803531] Lustre: lustre-MDT0001: new disk, initializing [12940.955546] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12941.020785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12941.030573] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12946.103831] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [12956.510465] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12963.291302] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12971.441341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12971.794714] Lustre: lustre-OST0000: new disk, initializing [12971.805797] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12971.825484] Lustre: 351450:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12971.977175] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12977.224911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12977.235030] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12977.345420] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12978.238926] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [12987.956513] Lustre: DEBUG MARKER: oleg625-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13028.843128] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13028.849763] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13028.864046] Lustre: Skipped 1 previous similar message [13028.867402] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13028.884611] Lustre: Skipped 1 previous similar message [13034.125520] Lustre: server umount lustre-OST0000 complete [13038.051600] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13038.056666] LustreError: Skipped 1 previous similar message [13038.059039] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13038.065101] Lustre: Skipped 1 previous similar message [13038.068798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13038.071553] Lustre: Skipped 2 previous similar messages [13044.490573] Lustre: server umount lustre-MDT0000 complete [13048.729645] LustreError: 349471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787473994 with bad export cookie 14672980647491151222 [13048.747078] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13048.759492] LustreError: 349471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13049.224581] Lustre: server umount lustre-MDT0001 complete [13064.099173] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing unload_modules_local [13067.636427] Key type lgssc unregistered [13068.105670] LNet: 353630:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13068.122071] LNetError: 353630:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13068.152863] LNet: Removed LNI 192.168.206.125@tcp [13069.382920] Key type .llcrypt unregistered [13069.385885] Key type ._llcrypt unregistered [13167.404581] Key type ._llcrypt registered [13167.407368] Key type .llcrypt registered [13167.519785] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing unload_modules_local [13167.810837] Key type .llcrypt unregistered [13167.819668] Key type ._llcrypt unregistered [13236.803238] Key type ._llcrypt registered [13236.805368] Key type .llcrypt registered [13236.889141] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing unload_modules_local [13237.186491] Key type .llcrypt unregistered [13237.190938] Key type ._llcrypt unregistered [13262.986086] Key type ._llcrypt registered [13262.988479] Key type .llcrypt registered [13263.066349] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing unload_modules_local [13263.325387] Key type .llcrypt unregistered [13263.328528] Key type ._llcrypt unregistered [13292.043775] Key type ._llcrypt registered [13292.046159] Key type .llcrypt registered [13292.141290] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing unload_modules_local [13292.401297] Key type .llcrypt unregistered [13292.405286] Key type ._llcrypt unregistered [13362.926448] Key type ._llcrypt registered [13362.928504] Key type .llcrypt registered [13363.004710] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing unload_modules_local [13363.227566] Key type .llcrypt unregistered [13363.230519] Key type ._llcrypt unregistered [13394.514368] Key type ._llcrypt registered [13394.516587] Key type .llcrypt registered [13394.611853] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing unload_modules_local [13394.893261] Key type .llcrypt unregistered [13394.897681] Key type ._llcrypt unregistered [13423.436614] Key type ._llcrypt registered [13423.438885] Key type .llcrypt registered [13423.571046] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing unload_modules_local [13423.812618] Key type .llcrypt unregistered [13423.815988] Key type ._llcrypt unregistered [13450.227083] Key type ._llcrypt registered [13450.230068] Key type .llcrypt registered [13450.336327] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing unload_modules_local [13450.680211] Key type .llcrypt unregistered [13450.681737] Key type ._llcrypt unregistered [13480.519289] Key type ._llcrypt registered [13480.521433] Key type .llcrypt registered [13480.637456] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing unload_modules_local [13480.959561] Key type .llcrypt unregistered [13480.961306] Key type ._llcrypt unregistered [13506.086577] Key type ._llcrypt registered [13506.089205] Key type .llcrypt registered [13506.179873] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing unload_modules_local [13506.463317] Key type .llcrypt unregistered [13506.467211] Key type ._llcrypt unregistered [13562.891470] Key type ._llcrypt registered [13562.893623] Key type .llcrypt registered [13563.021530] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing unload_modules_local [13563.343275] Key type .llcrypt unregistered [13563.347862] Key type ._llcrypt unregistered [13591.967787] Key type ._llcrypt registered [13591.971566] Key type .llcrypt registered [13592.073954] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing unload_modules_local [13592.397649] Key type .llcrypt unregistered [13592.400122] Key type ._llcrypt unregistered [13620.147969] Key type ._llcrypt registered [13620.152330] Key type .llcrypt registered [13620.305365] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing unload_modules_local [13620.547570] Key type .llcrypt unregistered [13620.551976] Key type ._llcrypt unregistered [13674.818261] Key type ._llcrypt registered [13674.819857] Key type .llcrypt registered [13674.963486] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_hostid [13689.642678] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing load_modules_local [13691.016574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13691.074890] alg: No test for adler32 (adler32-zlib) [13692.222178] Lustre: Lustre: Build Version: 2.17.57_65_gc740cad [13692.495605] LNet: Added LNI 192.168.206.125@tcp [8/256/0/180] [13694.207322] Key type lgssc registered [13695.382347] Lustre: Echo OBD driver; http://www.lustre.org/ [13732.310553] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13732.342254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13733.607479] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13733.645652] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13733.717927] Lustre: tst250fs-MDT0000: new disk, initializing [13733.826372] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13733.838358] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13738.528378] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [13748.407957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13748.516585] Lustre: 373873:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13748.556445] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13748.561081] Lustre: Skipped 1 previous similar message [13748.630414] Lustre: tst250fs-MDT0001: new disk, initializing [13748.694547] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13748.765680] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13748.771981] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13752.679916] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [13757.477523] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13764.566162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13764.889884] Lustre: tst250fs-OST0000: new disk, initializing [13764.896916] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13764.911235] Lustre: 375505:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13765.058893] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13772.249404] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [13774.441646] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13774.460486] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13774.621651] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13783.875913] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13784.086501] Lustre: tst250fs-OST0001: new disk, initializing [13784.092845] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13784.101417] Lustre: 376375:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13784.231293] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13789.791795] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13789.805807] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13789.882339] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13790.284218] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_default_debug -1 all [13795.268199] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13797.757541] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [13797.764058] Lustre: Skipped 2 previous similar messages [13820.390584] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13820.397341] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13820.407571] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13820.899597] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13820.916573] Lustre: Skipped 2 previous similar messages [13820.920362] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13820.925264] Lustre: Skipped 2 previous similar messages [13822.245832] Lustre: server umount tst250fs-MDT0000 complete [13825.659377] LustreError: 376374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787474771 with bad export cookie 8526554165204957750 [13825.666725] LustreError: MGC192.168.206.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13825.674227] LustreError: 376374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13825.952338] Lustre: server umount tst250fs-MDT0001 complete [13839.394390] Lustre: server umount tst250fs-OST0000 complete [13842.335213] Lustre: 372098:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787474771/real 1787474771] req@ffff9afc3cd2f100 x1874303006078976/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787474787 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13842.384223] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13842.982688] Lustre: server umount tst250fs-OST0001 complete [13860.204732] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_hostid [13867.886651] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing load_modules_local [13897.375664] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 04:47:20 (1787474840) [13899.237416] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13915.189112] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing set_hostid [13925.276062] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing load_modules_local [13956.510495] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13690 sec ========= 04:48:19 (1787474899) [13958.042935] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 04:48:21 (1787474901) === [13961.284382] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 04:48:24 (1787474904) === [14003.432688] Lustre: DEBUG MARKER: oleg625-server.virtnet: executing unload_modules_local [14006.034300] Key type lgssc unregistered [14006.424550] LNet: 385504:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14006.444813] LNetError: 385504:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14006.471964] LNet: Removed LNI 192.168.206.125@tcp [14007.639251] Key type .llcrypt unregistered [14007.641914] Key type ._llcrypt unregistered