[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 490829921 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003376] x2apic enabled [ 0.004011] Switched APIC routing to physical x2apic. [ 0.006015] kvm-guest: setup PV IPIs [ 0.009334] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.010000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.011020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.012013] pid_max: default: 32768 minimum: 301 [ 0.013142] LSM: Security Framework initializing [ 0.014057] Yama: becoming mindful. [ 0.015000] SELinux: Initializing. [ 0.015075] *** VALIDATE selinux *** [ 0.023727] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028426] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029134] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030115] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031144] *** VALIDATE tmpfs *** [ 0.032047] *** VALIDATE proc *** [ 0.034193] *** VALIDATE cgroup *** [ 0.035010] *** VALIDATE cgroup2 *** [ 0.037282] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038151] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039014] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040024] Spectre V2 : User space: Vulnerable [ 0.041016] Speculative Store Bypass: Vulnerable [ 0.044587] debug: unmapping init [mem 0xffffffffb5a59000-0xffffffffb5a60fff] [ 0.046184] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047743] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048028] ... version: 2 [ 0.049013] ... bit width: 48 [ 0.050016] ... generic registers: 4 [ 0.051015] ... value mask: 0000ffffffffffff [ 0.052016] ... max period: 00007fffffffffff [ 0.053015] ... fixed-purpose events: 3 [ 0.054014] ... event mask: 000000070000000f [ 0.055293] rcu: Hierarchical SRCU implementation. [ 0.057516] smp: Bringing up secondary CPUs ... [ 0.058664] x86: Booting SMP configuration: [ 0.059035] .... node #0, CPUs: #1 #2 #3 [ 0.062691] smp: Brought up 1 node, 4 CPUs [ 0.064013] smpboot: Max logical packages: 1 [ 0.065019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.140000] node 0 deferred pages initialised in 71ms [ 0.143208] devtmpfs: initialized [ 0.144287] x86/mm: Memory block size: 128MB [ 0.146656] gcov: version magic: 0x41383552 [ 0.147654] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.150073] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.153299] pinctrl core: initialized pinctrl subsystem [ 0.154155] [ 0.154596] ************************************************************* [ 0.157011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.158009] ** ** [ 0.160010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.162011] ** ** [ 0.164011] ** This means that this kernel is built to expose internal ** [ 0.166011] ** IOMMU data structures, which may compromise security on ** [ 0.167009] ** your system. ** [ 0.169014] ** ** [ 0.171011] ** If you see this message and you are not debugging the ** [ 0.173012] ** kernel, report this immediately to your vendor! ** [ 0.174009] ** ** [ 0.176011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.178012] ************************************************************* [ 0.179627] NET: Registered protocol family 16 [ 0.181424] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.183056] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.185066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.188117] cpuidle: using governor menu [ 0.190159] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.191409] PCI: Using configuration type 1 for base access [ 0.192146] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.199065] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.201024] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.205093] cryptd: max_cpu_qlen set to 1000 [ 0.208313] ACPI: Added _OSI(Module Device) [ 0.210014] ACPI: Added _OSI(Processor Device) [ 0.212029] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.214017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.218276] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.225384] ACPI: Interpreter enabled [ 0.227068] ACPI: PM: (supports S0 S3 S4 S5) [ 0.229014] ACPI: Using IOAPIC for interrupt routing [ 0.231156] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.234375] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.246224] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.249048] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.253080] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.259177] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.267494] acpiphp: Slot [2] registered [ 0.269139] acpiphp: Slot [5] registered [ 0.271166] acpiphp: Slot [6] registered [ 0.273155] acpiphp: Slot [7] registered [ 0.275163] acpiphp: Slot [8] registered [ 0.277143] acpiphp: Slot [9] registered [ 0.278000] acpiphp: Slot [10] registered [ 0.278000] acpiphp: Slot [3] registered [ 0.280106] acpiphp: Slot [4] registered [ 0.282116] acpiphp: Slot [11] registered [ 0.283099] acpiphp: Slot [12] registered [ 0.285120] acpiphp: Slot [13] registered [ 0.287142] acpiphp: Slot [14] registered [ 0.288098] acpiphp: Slot [15] registered [ 0.290114] acpiphp: Slot [16] registered [ 0.291090] acpiphp: Slot [17] registered [ 0.293095] acpiphp: Slot [18] registered [ 0.294106] acpiphp: Slot [19] registered [ 0.296106] acpiphp: Slot [20] registered [ 0.298133] acpiphp: Slot [21] registered [ 0.299124] acpiphp: Slot [22] registered [ 0.301195] acpiphp: Slot [23] registered [ 0.302104] acpiphp: Slot [24] registered [ 0.304134] acpiphp: Slot [25] registered [ 0.306106] acpiphp: Slot [26] registered [ 0.308150] acpiphp: Slot [27] registered [ 0.309103] acpiphp: Slot [28] registered [ 0.311129] acpiphp: Slot [29] registered [ 0.313096] acpiphp: Slot [30] registered [ 0.315142] acpiphp: Slot [31] registered [ 0.317070] PCI host bridge to bus 0000:00 [ 0.319024] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.323027] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.326026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.329023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.332025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.335030] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.338183] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.343164] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.347635] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.356797] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.362964] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.366020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.368017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.371018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.375634] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.377818] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.380037] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.382841] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.389017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.404020] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.409016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.414899] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.423016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.429031] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.447015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.456887] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.465017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.471018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.493016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.502198] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.509017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.514016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.541015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.550514] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.557015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.563016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.575016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.584203] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.591014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.595016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.608016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.620130] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.625016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.630022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.645020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.654119] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.657460] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.660385] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.663378] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.665311] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.671166] iommu: Default domain type: Passthrough [ 0.673504] SCSI subsystem initialized [ 0.675159] ACPI: bus type USB registered [ 0.676124] usbcore: registered new interface driver usbfs [ 0.679124] usbcore: registered new interface driver hub [ 0.681091] usbcore: registered new device driver usb [ 0.683173] pps_core: LinuxPPS API ver. 1 registered [ 0.684009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.688071] PTP clock support registered [ 0.690072] EDAC MC: Ver: 3.0.0 [ 0.692103] PCI: Using ACPI for IRQ routing [ 0.694774] NetLabel: Initializing [ 0.695008] NetLabel: domain hash size = 128 [ 0.697010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.699089] NetLabel: unlabeled traffic allowed by default [ 0.701128] vgaarb: loaded [ 0.702306] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.704016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.711675] clocksource: Switched to clocksource kvm-clock [ 0.820415] VFS: Disk quotas dquot_6.6.0 [ 0.821536] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.824124] *** VALIDATE ramfs *** [ 0.825389] *** VALIDATE hugetlbfs *** [ 0.826979] pnp: PnP ACPI init [ 0.829196] pnp: PnP ACPI: found 6 devices [ 0.857541] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.860899] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.863154] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.865231] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.867641] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.870185] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.872297] NET: Registered protocol family 2 [ 0.874396] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.879133] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.882047] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.887693] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.890890] TCP: Hash tables configured (established 65536 bind 65536) [ 0.893814] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.897372] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.900377] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.903527] NET: Registered protocol family 1 [ 0.906309] RPC: Registered named UNIX socket transport module. [ 0.908644] RPC: Registered udp transport module. [ 0.910554] RPC: Registered tcp transport module. [ 0.912381] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.915018] NET: Registered protocol family 44 [ 0.916775] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.919105] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.921427] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.924022] PCI: CLS 0 bytes, default 64 [ 0.926827] Unpacking initramfs... [ 2.329482] debug: unmapping init [mem 0xffff8c4d7cc54000-0xffff8c4d7ffbffff] [ 2.333760] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.336509] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.339753] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.859679] Initialise system trusted keyrings [ 2.861403] Key type blacklist registered [ 2.863323] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.872674] zbud: loaded [ 2.876071] *** VALIDATE nfs *** [ 2.877450] *** VALIDATE nfs4 *** [ 2.879157] pstore: using deflate compression [ 2.884427] Platform Keyring initialized [ 3.000366] NET: Registered protocol family 38 [ 3.002211] Key type asymmetric registered [ 3.003842] Asymmetric key parser 'x509' registered [ 3.005779] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.008712] io scheduler mq-deadline registered [ 3.010574] io scheduler kyber registered [ 3.012295] io scheduler bfq registered [ 3.015371] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.018675] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.021770] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.024812] ACPI: Power Button [PWRF] [ 3.030496] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.037367] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.054313] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.064321] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.101174] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.129558] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.157504] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.163324] Non-volatile memory driver v1.3 [ 3.164590] Linux agpgart interface v0.103 [ 3.196844] virtio_blk virtio1: [vda] 147232 512-byte logical blocks (75.4 MB/71.9 MiB) [ 3.200178] vda: detected capacity change from 0 to 75382784 [ 3.219652] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.222996] vdb: detected capacity change from 0 to 1073741824 [ 3.238388] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.241593] vdc: detected capacity change from 0 to 2621440000 [ 3.254730] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.257699] vdd: detected capacity change from 0 to 2621440000 [ 3.270802] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.273722] vde: detected capacity change from 0 to 4294967296 [ 3.295974] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.299070] vdf: detected capacity change from 0 to 4294967296 [ 3.312680] libphy: Fixed MDIO Bus: probed [ 3.322255] usbcore: registered new interface driver usbserial_generic [ 3.324358] usbserial: USB Serial support registered for generic [ 3.326600] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.330246] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.332222] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.333888] mousedev: PS/2 mouse device common for all mice [ 3.336299] rtc_cmos 00:05: RTC can wake from S4 [ 3.339498] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.340448] rtc_cmos 00:05: registered as rtc0 [ 3.346069] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.349393] intel_pstate: CPU model not supported [ 3.352655] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.354293] hid: raw HID events driver (C) Jiri Kosina [ 3.360383] usbcore: registered new interface driver usbhid [ 3.362769] usbhid: USB HID core driver [ 3.363056] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.364300] drop_monitor: Initializing network drop monitor service [ 3.371604] Initializing XFRM netlink socket [ 3.373438] NET: Registered protocol family 10 [ 3.376384] Segment Routing with IPv6 [ 3.378144] NET: Registered protocol family 17 [ 3.379710] mpls_gso: MPLS GSO support [ 3.386104] RAS: Correctable Errors collector initialized. [ 3.388547] AVX version of gcm_enc/dec engaged. [ 3.390372] AES CTR mode by8 optimization enabled [ 3.463806] sched_clock: Marking stable (3463719868, 0)->(4430819549, -967099681) [ 3.467404] registered taskstats version 1 [ 3.469655] Loading compiled-in X.509 certificates [ 3.471865] zswap: loaded using pool lzo/zbud [ 3.494937] Key type big_key registered [ 3.506748] Key type encrypted registered [ 3.508523] ima: No TPM chip found, activating TPM-bypass! [ 3.510716] ima: Allocated hash algorithm: sha1 [ 3.512752] ima: No architecture policies found [ 3.514757] evm: Initialising EVM extended attributes: [ 3.516744] evm: security.selinux [ 3.517821] evm: security.ima [ 3.519077] evm: security.capability [ 3.520345] evm: HMAC attrs: 0x1 [ 3.524145] rtc_cmos 00:05: setting system clock to 2026-09-11 10:44:34 UTC (1789123474) [ 3.530490] debug: unmapping init [mem 0xffffffffb6a03000-0xffffffffb6bfffff] [ 3.533823] debug: unmapping init [mem 0xffffffffb5782000-0xffffffffb5a58fff] [ 3.543071] Write protecting the kernel read-only data: 28672k [ 3.546741] debug: unmapping init [mem 0xffffffffb3e03000-0xffffffffb3ffffff] [ 3.549891] debug: unmapping init [mem 0xffffffffb4714000-0xffffffffb47fffff] [ 3.584874] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.595445] systemd[1]: Detected virtualization kvm. [ 3.597748] systemd[1]: Detected architecture x86-64. [ 3.600040] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.626952] systemd[1]: No hostname configured. [ 3.628889] systemd[1]: Set hostname to . [ 3.632107] random: systemd: uninitialized urandom read (16 bytes read) [ 3.634631] systemd[1]: Initializing machine ID from random generator. [ 3.669745] random: ln: uninitialized urandom read (6 bytes read) [ 3.760885] random: systemd: uninitialized urandom read (16 bytes read) [ 3.763837] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.768243] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.772760] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. [ OK ] Reached target Swap. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Reached target Sockets. Starting Journal Service... Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.397388] device-mapper: uevent: version 1.0.3 [ 4.400781] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.216294] virtio_net virtio0 ens2: renamed from eth0 [ 5.221576] random: fast init done [ 5.343125] scsi host0: ata_piix [ 5.408358] scsi host1: ata_piix [ 5.410381] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.413933] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.222323] dracut-initqueue[581]: RTNETLINK answers: File exists [ 10.028040] random: crng init done [ 10.029863] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.673791] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.876949] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.141674] SELinux: Disabled at runtime. [ 12.208291] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.216518] systemd[1]: Detected virtualization kvm. [ 12.218657] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.757210] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.761451] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.766902] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.771278] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.775217] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.786503] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.797738] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on RPCbind Server Activation Socket. [ 12.883543] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Mounting Huge Pages File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Created slice system-getty.slice. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.274492] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.569556] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.577829] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.702620] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.715201] EDAC sbridge: Ver: 1.1.2 [ 15.569287] Key type dns_resolver registered [ 15.885990] NFS: Registering the id_resolver key type [ 15.888253] Key type id_resolver registered [ 15.889942] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started OpenSSH server daemon. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg446-server login: [ 72.344419] libcfs: loading out-of-tree module taints kernel. [ 72.430631] Key type ._llcrypt registered [ 72.432321] Key type .llcrypt registered [ 72.624905] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 93.956269] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 96.346769] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 96.363953] alg: No test for adler32 (adler32-zlib) [ 97.974942] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 99.346134] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 101.152232] Key type lgssc registered [ 103.436390] Lustre: Echo OBD driver; http://www.lustre.org/ [ 122.157017] hrtimer: interrupt took 2127300 ns [ 124.277186] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 166.687596] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 181.419379] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 181.494708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 182.815915] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 182.852055] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 182.981230] Lustre: lustre-MDT0000: new disk, initializing [ 183.097482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 183.125275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 188.037488] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 201.412209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 201.545090] Lustre: 6515:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 201.590148] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 201.597621] Lustre: Skipped 1 previous similar message [ 201.719457] Lustre: lustre-MDT0001: new disk, initializing [ 201.810044] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 201.866274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 201.878990] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 206.595369] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 212.404522] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 224.051452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 224.592393] Lustre: lustre-OST0000: new disk, initializing [ 224.608664] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 224.623966] Lustre: 8453:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 224.751202] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 232.027210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 232.048209] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 232.176528] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 232.575385] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 248.879759] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 249.003906] Lustre: lustre-OST0001: new disk, initializing [ 249.009592] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 249.018450] Lustre: 9524:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 249.115109] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 255.036897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 255.061556] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 255.218955] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 256.650791] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 269.447846] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 280.720241] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 287.166531] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing check_logdir /tmp/testlogs/ [ 293.147409] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing yml_node [ 297.994456] Lustre: DEBUG MARKER: Client: 2.17.57.138 [ 300.706209] Lustre: DEBUG MARKER: MDS: 2.17.57.138 [ 303.250719] Lustre: DEBUG MARKER: OSS: 2.17.57.138 [ 305.077150] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Sep 11 06:49:33 EDT 2026 [ 322.002687] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 323.824538] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 339.943537] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 339.958323] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 339.991128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 342.501747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 342.504489] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 347.620272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 347.628447] Lustre: Skipped 6 previous similar messages [ 352.740581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 352.748448] Lustre: Skipped 3 previous similar messages [ 354.272306] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 354.589125] Lustre: server umount lustre-MDT0000 complete [ 358.143222] LustreError: 10340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789123829 with bad export cookie 14880908466649897574 [ 358.147273] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 358.163308] LustreError: 10340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 362.977090] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 362.979221] LustreError: 6521:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 362.979827] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 362.993064] Lustre: Skipped 3 previous similar messages [ 363.061053] LustreError: 6521:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 364.001712] LustreError: 6522:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 364.027256] LustreError: 6522:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 364.692276] Lustre: server umount lustre-MDT0001 complete [ 368.877078] Lustre: server umount lustre-OST0000 complete [ 372.951988] Lustre: server umount lustre-OST0001 complete [ 380.350927] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 389.545957] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 427.763476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 428.006125] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 428.036399] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 428.119410] Lustre: lustre-MDT0000: new disk, initializing [ 428.210619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 428.242580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 433.418597] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 445.439127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 445.607798] Lustre: 15077:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 445.706953] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 445.719214] Lustre: Skipped 1 previous similar message [ 445.859373] Lustre: lustre-MDT0001: new disk, initializing [ 446.009897] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 446.085312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 446.095881] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 451.571148] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 461.558380] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 467.668640] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 474.758193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 475.035963] Lustre: lustre-OST0000: new disk, initializing [ 475.044157] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 475.062509] Lustre: 17047:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 475.173784] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 476.623766] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 476.635637] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 476.799883] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 481.962540] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 492.319714] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 497.757945] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 498.047363] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 503.389277] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 503.642473] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 507.362699] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 507.383950] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 507.413215] Lustre: Skipped 1 previous similar message [ 507.441892] Lustre: Skipped 4 previous similar messages [ 512.257191] Lustre: server umount lustre-OST0000 complete [ 517.601772] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 517.623666] Lustre: Skipped 1 previous similar message [ 523.026185] Lustre: server umount lustre-MDT0000 complete [ 527.016217] LustreError: 15070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789123997 with bad export cookie 14880908466649899107 [ 527.024296] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 527.028106] LustreError: 15070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 527.269257] Lustre: server umount lustre-MDT0001 complete [ 533.710480] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 06:53:22 (1789124002) [ 554.529180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 555.221572] LustreError: 19924:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 555.324941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 560.609171] LustreError: 19925:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 561.215541] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 565.731367] LustreError: 19924:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 571.227753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 578.216974] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 587.121518] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 596.286822] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 599.548569] Lustre: Failing over lustre-MDT0000 [ 599.978340] Lustre: server umount lustre-MDT0000 complete [ 601.570540] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 601.589426] LustreError: 19925:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 601.598644] Lustre: Skipped 1 previous similar message [ 601.645063] LustreError: 19925:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 617.952137] Lustre: 3648:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789124072/real 1789124072] req@ffff8c4cc82ca940 x1876032036705408/t0(0) o400->MGC192.168.204.146@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789124088 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 617.994415] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 618.035982] LustreError: 19924:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 618.067068] LustreError: 19924:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [ 621.364964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 628.195221] LustreError: 3646:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8c4cc834e1c0 x1876032036708224/t0(0) o250->MGC192.168.204.146@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 628.438807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 628.443059] Lustre: Skipped 1 previous similar message [ 628.477423] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 633.689831] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 633.838883] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 633.850078] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 633.882838] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 644.098807] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 649.187746] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 649.200436] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 649.221591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 649.237841] Lustre: Skipped 4 previous similar messages [ 654.256569] Lustre: server umount lustre-MDT0000 complete [ 656.355448] LustreError: 19925:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 656.377198] LustreError: 19925:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 659.024606] LustreError: 19905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789124129 with bad export cookie 14880908466649900521 [ 659.030867] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 659.051684] LustreError: 19905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 659.876430] Lustre: server umount lustre-MDT0001 complete [ 668.855549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 674.440658] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 684.942648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 689.675452] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 698.799434] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 705.480886] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 707.785840] Lustre: Failing over lustre-MDT0000 [ 708.150020] Lustre: server umount lustre-MDT0000 complete [ 710.884506] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 710.908643] Lustre: Skipped 2 previous similar messages [ 721.377859] LustreError: 23263:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 721.404799] LustreError: 23263:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 9 previous similar messages [ 726.049828] Lustre: 3648:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789124181/real 1789124181] req@ffff8c4df7684000 x1876032036736128/t0(0) o400->MGC192.168.204.146@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789124197 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 726.077557] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 726.831505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 736.363404] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xce839b6871c52993 [ 736.379283] Lustre: MGC192.168.204.146@tcp: Connection restored to 0@lo (at 0@lo) [ 736.388455] Lustre: Skipped 1 previous similar message [ 736.809435] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 741.871716] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 741.880055] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 741.920824] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 742.054825] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 753.152686] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 757.218816] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 757.237338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 757.243891] Lustre: Skipped 2 previous similar messages [ 763.599837] Lustre: server umount lustre-MDT0000 complete [ 768.230348] LustreError: 23244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789124239 with bad export cookie 14880908466649901459 [ 768.243252] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 768.250659] LustreError: 23244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 768.640644] Lustre: server umount lustre-MDT0001 complete [ 779.620723] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 06:57:27 (1789124247) [ 798.347559] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 810.159108] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 850.942189] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 861.777946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 862.019932] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 862.052805] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 862.135590] Lustre: lustre-MDT0000: new disk, initializing [ 862.188038] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 862.199150] Lustre: Skipped 3 previous similar messages [ 862.208903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 866.897908] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 877.471785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 877.626461] Lustre: 29637:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 877.925191] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 882.703490] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 893.349309] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 900.265427] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 907.606347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 907.875941] Lustre: 31607:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 909.455903] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 909.490863] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 909.608319] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 909.641153] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 913.672318] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 924.296505] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 930.183973] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 955.883934] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 955.887307] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 955.903050] LustreError: Skipped 1 previous similar message [ 955.919390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 955.928546] Lustre: Skipped 3 previous similar messages [ 955.963223] Lustre: Skipped 3 previous similar messages [ 958.705389] Lustre: server umount lustre-OST0000 complete [ 965.089322] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 971.138653] Lustre: server umount lustre-MDT0000 complete [ 971.242491] LustreError: 29650:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 971.273375] LustreError: 29650:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 11 previous similar messages [ 976.293714] LustreError: 29631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789124447 with bad export cookie 14880908466649901872 [ 976.307293] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 976.307825] LustreError: 29631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 977.236509] Lustre: server umount lustre-MDT0001 complete [ 992.777743] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 996.055487] Key type lgssc unregistered [ 996.326216] LNet: 33608:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 996.331625] LNetError: 33608:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 996.356313] LNet: Removed LNI 192.168.204.146@tcp [ 997.096254] Key type .llcrypt unregistered [ 997.098069] Key type ._llcrypt unregistered [ 1023.151417] Key type ._llcrypt registered [ 1023.153713] Key type .llcrypt registered [ 1023.279303] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 1024.584802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1024.658931] alg: No test for adler32 (adler32-zlib) [ 1025.747429] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 1026.009080] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 1027.672197] Key type lgssc registered [ 1028.888653] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1037.628245] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1037.654630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1039.306329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1044.415199] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1049.572587] LustreError: 35025:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1049.591021] LustreError: 35025:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1052.558091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1052.923163] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1057.664675] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1066.706783] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1073.370742] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1080.765645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1081.201392] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1087.883784] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1095.838313] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1097.594590] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1110.215571] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 07:02:58 (1789124578) [ 1112.214441] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1114.272274] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1122.047925] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 07:03:10 (1789124590) [ 1131.649930] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 07:03:19 (1789124599) [ 1132.995927] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1143.362303] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1143.367505] Lustre: Skipped 2 previous similar messages [ 1145.803994] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 07:03:33 (1789124613) [ 1148.453904] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1148.457740] Lustre: Skipped 2 previous similar messages [ 1157.453407] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 1157.463253] Lustre: Skipped 1 previous similar message [ 1174.264550] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1174.274042] Lustre: Skipped 5 previous similar messages [ 1177.671299] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 07:04:05 (1789124645) [ 1180.418576] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1184.744139] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1194.101772] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 07:04:22 (1789124662) [ 1207.348257] Lustre: Setting parameter general.jobid_name=jobid_name=jobid_name=%e.root in log params [ 1207.355609] Lustre: Skipped 7 previous similar messages [ 1209.363578] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 07:04:37 (1789124677) [ 1225.699420] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1225.703716] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1225.734212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1230.239936] Lustre: server umount lustre-OST0000 complete [ 1235.940309] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1235.947980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1235.964175] Lustre: Skipped 2 previous similar messages [ 1235.974206] Lustre: Skipped 2 previous similar messages [ 1241.057772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1241.065338] Lustre: Skipped 1 previous similar message [ 1245.154164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1245.166945] Lustre: Skipped 1 previous similar message [ 1248.224238] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1248.547481] Lustre: server umount lustre-MDT0000 complete [ 1250.273675] LustreError: 35025:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1250.295238] LustreError: 35025:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1252.949440] LustreError: 37945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789124723 with bad export cookie 15375448161818501307 [ 1252.959468] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1252.972723] LustreError: 37945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1253.473051] Lustre: server umount lustre-MDT0001 complete [ 1267.920960] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 1271.545304] Key type lgssc unregistered [ 1271.945266] LNet: 42893:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1271.962595] LNetError: 42893:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1271.988703] LNet: Removed LNI 192.168.204.146@tcp [ 1272.807899] Key type .llcrypt unregistered [ 1272.814274] Key type ._llcrypt unregistered [ 1293.548416] Key type ._llcrypt registered [ 1293.550548] Key type .llcrypt registered [ 1293.653449] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 1309.262799] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 1310.330040] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1310.345238] alg: No test for adler32 (adler32-zlib) [ 1311.434938] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 1311.731837] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 1313.449057] Key type lgssc registered [ 1314.777296] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1351.998393] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1352.035937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1353.331496] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1353.376453] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1353.511889] Lustre: lustre-MDT0000: new disk, initializing [ 1353.638749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1353.662118] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1358.529276] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1370.413045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1370.583342] Lustre: 46048:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1370.678685] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1370.687853] Lustre: Skipped 1 previous similar message [ 1370.759423] Lustre: lustre-MDT0001: new disk, initializing [ 1370.906777] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1370.974608] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1370.997283] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1377.013384] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1389.323802] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1398.260615] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1408.258327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1408.871233] Lustre: lustre-OST0000: new disk, initializing [ 1408.879661] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1408.885854] Lustre: 48018:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1409.112653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1417.120988] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1417.303668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1417.328909] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1417.625974] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1428.754783] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1434.584710] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1434.907271] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1441.380291] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1441.647709] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1448.428941] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1448.431258] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1448.435760] LustreError: Skipped 1 previous similar message [ 1448.468468] Lustre: Skipped 1 previous similar message [ 1448.470766] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1453.538478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1453.554264] Lustre: Skipped 2 previous similar messages [ 1458.145685] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1458.360393] Lustre: server umount lustre-OST0000 complete [ 1463.273845] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1463.300087] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1463.345558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1468.903548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1468.910798] Lustre: Skipped 1 previous similar message [ 1469.015524] Lustre: server umount lustre-MDT0000 complete [ 1473.107190] LustreError: 46041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789124944 with bad export cookie 11290468332414325569 [ 1473.119754] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1473.129362] LustreError: 46041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1473.668840] Lustre: server umount lustre-MDT0001 complete [ 1484.882616] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 07:09:13 (1789124953) [ 1491.431702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1492.051198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1497.060461] LustreError: 50482:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1497.069376] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1497.085861] LustreError: 50482:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 1501.155678] LustreError: 50481:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1506.280682] LustreError: 50482:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1506.436226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1506.799698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1511.944912] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1520.398693] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1526.533088] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1533.895316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1534.490336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1541.953613] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1550.735328] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1559.209572] LustreError: 53010:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1560.066280] Lustre: Modifying parameter general.timeout=1 in log params [ 1561.812831] Lustre: Modifying parameter general.timeout=3 in log params [ 1561.817307] Lustre: Skipped 1 previous similar message [ 1564.658368] Lustre: Modifying parameter general.timeout=6 in log params [ 1564.662618] Lustre: Skipped 2 previous similar messages [ 1569.374172] Lustre: Modifying parameter general.timeout=11 in log params [ 1569.384582] Lustre: Skipped 4 previous similar messages [ 1578.071432] Lustre: Modifying parameter general.timeout=20 in log params [ 1578.081457] Lustre: Skipped 8 previous similar messages [ 1594.856702] Lustre: Modifying parameter general.timeout=36 in log params [ 1594.863415] Lustre: Skipped 15 previous similar messages [ 1627.699491] Lustre: Modifying parameter general.timeout=68 in log params [ 1627.716399] Lustre: Skipped 31 previous similar messages [ 1696.461329] Lustre: Disabling parameter general.timeout= in log params [ 1696.465645] Lustre: Skipped 61 previous similar messages [ 1700.541878] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 07:12:48 (1789125168) [ 1703.650635] Lustre: 59964:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1705.483350] Lustre: 60063:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1709.634846] Lustre: 60274:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1709.652564] Lustre: 60274:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1743.977700] Lustre: 61846:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1743.987754] Lustre: 61846:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1744.031635] LustreError: 61846:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1748.749218] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 07:13:37 (1789125217) [ 1762.785465] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1762.801064] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1762.813852] Lustre: Skipped 1 previous similar message [ 1762.820137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1762.827823] Lustre: Skipped 1 previous similar message [ 1766.393073] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1767.924889] Lustre: server umount lustre-MDT0000 complete [ 1771.489668] LustreError: 51811:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1771.510193] LustreError: 51811:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1771.770190] LustreError: 50462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789125242 with bad export cookie 11290468332414326451 [ 1771.772924] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1771.790820] LustreError: 50462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1772.239587] Lustre: server umount lustre-MDT0001 complete [ 1786.918775] Lustre: server umount lustre-OST0000 complete [ 1816.830082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1817.238251] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1817.625970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1822.651948] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1833.710706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1833.858426] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1833.910189] Lustre: 64378:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1839.374937] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1847.691913] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1854.569087] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1862.197318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1862.494380] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1863.031032] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1863.039066] Lustre: Skipped 1 previous similar message [ 1870.537633] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1879.268438] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1880.947976] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1880.957597] Lustre: Skipped 23 previous similar messages [ 1881.023199] Lustre: 66944:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1892.845434] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1892.854774] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1892.857642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1892.857650] Lustre: Skipped 3 previous similar messages [ 1892.874923] Lustre: Skipped 2 previous similar messages [ 1892.900449] LustreError: Skipped 1 previous similar message [ 1894.758793] Lustre: server umount lustre-OST0000 complete [ 1901.026023] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1901.046282] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1905.978866] Lustre: server umount lustre-MDT0000 complete [ 1908.196920] LustreError: 64385:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1908.225578] LustreError: 64385:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1910.115829] LustreError: 65129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789125381 with bad export cookie 11290468332414329629 [ 1910.118696] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1910.130420] LustreError: 65129:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1910.931703] Lustre: server umount lustre-MDT0001 complete [ 1927.286748] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 1931.242140] Key type lgssc unregistered [ 1931.599700] LNet: 68164:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1931.605827] LNetError: 68164:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1931.619908] LNet: Removed LNI 192.168.204.146@tcp [ 1932.693124] Key type .llcrypt unregistered [ 1932.695138] Key type ._llcrypt unregistered [ 1957.870445] Key type ._llcrypt registered [ 1957.875604] Key type .llcrypt registered [ 1958.021502] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 1959.007905] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1959.026842] alg: No test for adler32 (adler32-zlib) [ 1960.087680] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 1960.317441] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 1961.976232] Key type lgssc registered [ 1962.975713] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1970.193728] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1970.223587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1971.617705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1976.218692] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1981.924918] LustreError: 69577:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1981.943442] LustreError: 69577:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1984.819182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1985.880794] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1991.737083] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2003.234494] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2010.176236] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2017.791663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2018.431954] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2024.573211] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2032.618389] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2050.632420] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2062.306195] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2062.312781] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2062.324070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2062.825146] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2062.829883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2062.836229] Lustre: Skipped 1 previous similar message [ 2062.859913] Lustre: Skipped 1 previous similar message [ 2066.113675] Lustre: server umount lustre-MDT0000 complete [ 2067.938450] LustreError: 70309:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2067.965309] LustreError: 70309:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2069.965701] LustreError: 69557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789125540 with bad export cookie 11902399828025576420 [ 2069.972859] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2070.242931] Lustre: server umount lustre-MDT0001 complete [ 2084.545018] Lustre: server umount lustre-OST0000 complete [ 2110.503691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2110.829271] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2110.860717] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2111.153474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2116.625431] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2128.985591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2129.241245] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2129.306210] Lustre: 74474:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2130.107724] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2136.273567] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2144.796419] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2151.188068] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2159.311422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2159.450498] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2160.063346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2167.083368] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2175.505549] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2177.058561] Lustre: 77045:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2178.892313] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2184.169452] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2184.169708] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2184.191683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2184.214468] Lustre: Skipped 1 previous similar message [ 2189.302257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2189.307195] Lustre: Skipped 1 previous similar message [ 2190.333634] Lustre: server umount lustre-OST0000 complete [ 2196.456623] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2196.463770] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2196.480788] Lustre: Skipped 1 previous similar message [ 2196.487531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2201.880101] Lustre: server umount lustre-MDT0000 complete [ 2204.642041] LustreError: 74484:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2204.671645] LustreError: 74484:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2205.514640] LustreError: 74465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789125676 with bad export cookie 11902399828025577267 [ 2205.530252] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2205.535658] LustreError: 74465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2205.962849] Lustre: server umount lustre-MDT0001 complete [ 2216.898485] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 2219.962813] Key type lgssc unregistered [ 2220.437104] LNet: 78266:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2220.456200] LNetError: 78266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2220.475035] LNet: Removed LNI 192.168.204.146@tcp [ 2221.459879] Key type .llcrypt unregistered [ 2221.462403] Key type ._llcrypt unregistered [ 2254.901486] Key type ._llcrypt registered [ 2254.903386] Key type .llcrypt registered [ 2255.012621] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2256.177053] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2256.301757] alg: No test for adler32 (adler32-zlib) [ 2257.494875] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 2257.774564] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 2259.464174] Key type lgssc registered [ 2260.696667] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2268.186774] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2268.208246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2269.594481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2273.712943] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2279.907886] LustreError: 79682:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2279.953539] LustreError: 79682:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2282.287786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2283.164436] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2287.940027] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2297.386896] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2303.836838] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2310.630528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2311.240550] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2316.874242] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2324.788598] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2331.317933] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2332.148333] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2333.693320] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2333.702440] Lustre: Skipped 1 previous similar message [ 2336.257282] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2336.263225] Lustre: Skipped 2 previous similar messages [ 2340.700931] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2340.711827] Lustre: Skipped 4 previous similar messages [ 2348.902151] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2348.907296] Lustre: Skipped 8 previous similar messages [ 2365.797121] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2365.805702] Lustre: Skipped 16 previous similar messages [ 2397.975916] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2397.982543] Lustre: Skipped 29 previous similar messages [ 2436.524215] Lustre: 87785:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2439.717174] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 07:25:08 (1789125908) [ 2441.417128] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2443.454979] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 07:25:11 (1789125911) [ 2450.401652] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2450.405476] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2450.422456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2450.434803] Lustre: Skipped 1 previous similar message [ 2450.444256] Lustre: Skipped 1 previous similar message [ 2455.555528] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2455.561316] Lustre: Skipped 1 previous similar message [ 2460.645217] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2460.659447] Lustre: Skipped 1 previous similar message [ 2462.176190] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2462.505126] Lustre: server umount lustre-OST0000 complete [ 2470.886726] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2470.902791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2470.907000] Lustre: Skipped 1 previous similar message [ 2474.316804] Lustre: server umount lustre-MDT0000 complete [ 2476.003774] LustreError: 79679:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2476.035494] LustreError: 79679:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2478.829138] LustreError: 85591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789125949 with bad export cookie 16048453695798201024 [ 2478.838049] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2478.848153] LustreError: 85591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2479.559918] Lustre: server umount lustre-MDT0001 complete [ 2492.575217] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 2495.936756] Key type lgssc unregistered [ 2496.397687] LNet: 89042:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2496.409637] LNetError: 89042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2496.442599] LNet: Removed LNI 192.168.204.146@tcp [ 2497.321813] Key type .llcrypt unregistered [ 2497.326805] Key type ._llcrypt unregistered [ 2501.896859] Key type ._llcrypt registered [ 2501.898885] Key type .llcrypt registered [ 2502.007614] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2506.530979] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules [ 2507.418045] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2507.609049] alg: No test for adler32 (adler32-zlib) [ 2508.665281] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 2508.848043] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 2510.488325] Key type lgssc registered [ 2511.756364] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2512.866270] LustreError: 90245:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2529.706214] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2534.582720] LustreError: 90245:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2538.219589] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2538.239573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2539.570736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2543.365311] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2549.735045] LustreError: 91272:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2549.759910] LustreError: 91272:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2553.198552] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 07:27:01 (1789126021) [ 2554.862141] LustreError: 91271:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2559.108518] LustreError: 91303:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2559.133849] LustreError: 91303:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2559.155611] LustreError: 91303:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 2559.587803] Lustre: server umount lustre-MDT0000 complete [ 2574.770245] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 2578.053735] Key type lgssc unregistered [ 2578.311857] LNet: 92800:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2578.330384] LNetError: 92800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2578.349710] LNet: Removed LNI 192.168.204.146@tcp [ 2579.287741] Key type .llcrypt unregistered [ 2579.300262] Key type ._llcrypt unregistered [ 2598.499680] Key type ._llcrypt registered [ 2598.502563] Key type .llcrypt registered [ 2598.693647] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2600.273354] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2600.433064] alg: No test for adler32 (adler32-zlib) [ 2601.608562] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 2601.916787] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 2603.568300] Key type lgssc registered [ 2605.084609] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2613.850569] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2613.894501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2615.473303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2620.311024] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2626.020445] LustreError: 94022:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2626.043771] LustreError: 94022:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2629.994955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2630.756515] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2635.934856] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2645.241328] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2651.456440] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2659.110807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2659.755705] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2666.283664] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2674.776069] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2727.802183] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 07:29:56 (1789126196) [ 2730.357804] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2733.012190] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 07:30:00 (1789126200) [ 2741.731222] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2741.733905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2741.755477] Lustre: Skipped 2 previous similar messages [ 2741.764047] Lustre: Skipped 2 previous similar messages [ 2744.729925] Lustre: server umount lustre-MDT0000 complete [ 2746.862333] LustreError: 94021:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2746.874204] LustreError: 94021:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2749.311659] LustreError: 94002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789126220 with bad export cookie 12952620885728660917 [ 2749.319059] LustreError: 94002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2749.319905] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2749.773528] Lustre: server umount lustre-MDT0001 complete [ 2764.404201] Lustre: server umount lustre-OST0000 complete [ 2779.553364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2780.172545] LustreError: 98316:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2780.458649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2785.174797] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2785.766016] LustreError: 98317:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2790.891679] LustreError: 98316:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2795.714344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2796.005846] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2796.748412] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2801.656611] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2811.331118] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2818.320854] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2835.352494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2835.635862] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2835.650881] LustreError: 98304:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2835.708175] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2835.726391] LustreError: 100400:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2835.736595] LustreError: 100400:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2835.744080] LustreError: 100400:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2835.852637] Lustre: server umount lustre-OST0000 complete [ 2835.859261] LustreError: 100400:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2846.404266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2846.577563] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2846.607697] LustreError: 98306:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2846.683292] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2846.692498] LustreError: 100867:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2846.701960] LustreError: 100867:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2846.711463] LustreError: 100867:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2846.831896] Lustre: server umount lustre-OST0000 complete [ 2846.835206] LustreError: 100867:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2858.420251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2858.717924] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2858.724808] Lustre: Found index 0 for lustre-OST0000, updating log [ 2858.738982] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2858.876519] Lustre: lustre-OST0000: new disk, initializing [ 2858.884087] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2858.895790] Lustre: 101404:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2859.300880] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2866.624222] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2871.172032] Lustre: Failing over lustre-OST0000 [ 2871.317619] Lustre: server umount lustre-OST0000 complete [ 2878.434288] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2878.441707] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2878.455669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2880.481967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2880.485202] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2882.576607] Lustre: server umount lustre-MDT0000 complete [ 2885.603828] LustreError: 99047:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2885.627542] LustreError: 99047:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2887.054023] LustreError: 98297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789126358 with bad export cookie 12952620885728663668 [ 2887.063326] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2887.073831] LustreError: 98297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2897.660334] Lustre: server umount lustre-MDT0001 complete [ 2909.316338] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 07:32:57 (1789126377) [ 2929.378178] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2944.828957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2945.736023] LustreError: 103754:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2945.961156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2952.799938] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2963.437968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2971.148342] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2975.777735] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2984.393567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2984.924117] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2984.932063] Lustre: Skipped 1 previous similar message [ 2984.950409] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2989.053265] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2989.113705] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2989.129264] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2993.658061] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3003.553176] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3003.682606] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3003.834990] Lustre: lustre-OST0001: new disk, initializing [ 3003.841753] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3003.848216] Lustre: 106076:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3010.236024] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3010.256678] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3010.446455] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3012.966395] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3022.831781] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3029.175500] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3036.132578] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3036.134698] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3036.152654] Lustre: Skipped 1 previous similar message [ 3036.169401] Lustre: Skipped 3 previous similar messages [ 3040.563158] Lustre: server umount lustre-MDT0001 complete [ 3041.255680] LustreError: 103755:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3041.280839] LustreError: 103755:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 3053.204184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3053.283752] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3053.292080] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3053.385094] Lustre: 103742:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3054.402781] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3054.420959] Lustre: Skipped 1 previous similar message [ 3054.896144] LustreError: 107605:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3054.913774] LustreError: 107605:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3054.931616] LustreError: 103746:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3055.102334] LustreError: 107614:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.146@tcp: cfg command failed: rc = -17 [ 3055.109319] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3055.109319] [ 3059.687129] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3060.130801] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3072.535691] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 07:35:40 (1789126540) [ 3095.746523] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3118.902707] Lustre: 109689:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3147.023591] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3151.855685] Lustre: 110826:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3248.611576] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3248.622635] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3248.639790] Lustre: Skipped 1 previous similar message [ 3248.645996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3248.655630] Lustre: Skipped 1 previous similar message [ 3250.916305] Lustre: server umount lustre-MDT0000 complete [ 3254.249486] LustreError: 106555:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3254.264751] LustreError: 106555:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 12 previous similar messages [ 3255.808233] LustreError: 103734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789126726 with bad export cookie 12952620885728664669 [ 3255.808730] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3255.816145] LustreError: 103734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3256.105710] Lustre: server umount lustre-MDT0001 complete [ 3270.263894] Lustre: server umount lustre-OST0000 complete [ 3284.464101] Lustre: server umount lustre-OST0001 complete [ 3291.501107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3302.069547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3312.471421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3326.012663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3351.406573] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3362.918820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3362.936238] Lustre: lustre-MDT0000: reset Object Index mappings [ 3363.484304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3368.781691] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3378.937066] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3378.992513] Lustre: lustre-MDT0001: reset Object Index mappings [ 3385.534790] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3388.554967] Lustre: 115934:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3395.544439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3402.697873] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3404.402725] LustreError: 116288:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3404.423386] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:226 to 0x280000400:257) [ 3404.429226] LustreError: 116288:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 3409.532402] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 3411.585286] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3417.603815] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3417.614200] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3418.728616] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3427.229624] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3431.764622] Lustre: 117811:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3492.378507] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 07:42:40 (1789126960) [ 3517.924570] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3517.930614] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3517.942838] Lustre: Skipped 3 previous similar messages [ 3517.953567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3517.959418] Lustre: Skipped 4 previous similar messages [ 3531.232123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3531.425816] Lustre: server umount lustre-MDT0000 complete [ 3534.820242] LustreError: 114764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789127005 with bad export cookie 12952620885728711527 [ 3534.828309] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3534.840763] LustreError: 114764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3535.343562] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3535.371091] Lustre: Skipped 4 previous similar messages [ 3541.543347] Lustre: server umount lustre-MDT0001 complete [ 3551.656880] Lustre: server umount lustre-OST0000 complete [ 3561.791889] Lustre: server umount lustre-OST0001 complete [ 3569.429129] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 3580.259551] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3621.274764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3621.494714] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3621.529256] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3621.629772] Lustre: lustre-MDT0000: new disk, initializing [ 3621.755235] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3621.771293] Lustre: Skipped 3 previous similar messages [ 3621.810452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3626.933543] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3633.453559] Lustre: server umount lustre-MDT0000 complete [ 3643.765591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3644.112058] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3644.126940] Lustre: 122090:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3644.153056] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3648.447636] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3654.579490] Lustre: server umount lustre-MDT0000 complete [ 3665.261132] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 07:45:33 (1789127133) [ 3667.140460] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3669.745657] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 07:45:37 (1789127137) [ 3690.708644] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3701.472148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3706.492743] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3718.472896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3718.729853] Lustre: 124141:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3718.874812] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3718.881297] Lustre: Skipped 1 previous similar message [ 3719.012950] Lustre: lustre-MDT0001: new disk, initializing [ 3719.188175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3719.217409] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3725.175270] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3730.853520] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3738.674541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3739.076828] Lustre: lustre-OST0000: new disk, initializing [ 3739.080557] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3739.086895] Lustre: 125718:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3740.573196] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3740.593510] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3740.701175] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3746.055409] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3758.388088] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3758.605708] Lustre: lustre-OST0001: new disk, initializing [ 3758.610877] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3758.624846] Lustre: 126591:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3759.853041] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3759.863519] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3759.989458] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3765.616595] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3776.782108] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3780.817231] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4214.785589] Lustre: 133011:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4214.793570] Lustre: 133011:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4214.799834] Lustre: 133011:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4214.806205] Lustre: 133011:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4214.816458] Lustre: 133011:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4214.822283] Lustre: 133011:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4516.174664] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 07:59:43 (1789127983) [ 4533.227646] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4533.230761] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4533.231925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4533.231931] Lustre: Skipped 16 previous similar messages [ 4533.253352] Lustre: Skipped 1 previous similar message [ 4538.340445] LustreError: 124148:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4538.371898] LustreError: 124148:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 9 previous similar messages [ 4538.395958] Lustre: server umount lustre-MDT0000 complete [ 4542.722348] LustreError: 133058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789128013 with bad export cookie 12952620885728781905 [ 4542.730295] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4542.738995] LustreError: 133058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4543.460297] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4543.483641] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4543.506895] Lustre: Skipped 4 previous similar messages [ 4552.707729] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4552.717216] Lustre: Skipped 2 previous similar messages [ 4557.280137] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4557.773397] Lustre: server umount lustre-MDT0001 complete [ 4576.225442] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4576.408037] Lustre: server umount lustre-OST0000 complete [ 4586.636510] Lustre: server umount lustre-OST0001 complete [ 4594.464935] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 4606.278064] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 4646.379091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4646.723367] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4646.753956] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4646.868718] Lustre: lustre-MDT0000: new disk, initializing [ 4647.026220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4647.037707] Lustre: Skipped 5 previous similar messages [ 4647.069826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4653.157848] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4664.606499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4664.783360] Lustre: 136083:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4664.848868] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4664.852859] Lustre: Skipped 1 previous similar message [ 4665.009541] Lustre: lustre-MDT0001: new disk, initializing [ 4665.227298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4665.240278] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4671.163615] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4681.029243] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4686.758658] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4694.064171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4694.481540] Lustre: lustre-OST0000: new disk, initializing [ 4694.487507] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4694.495661] Lustre: 138052:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4696.436492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4696.453631] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4696.609885] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4700.979078] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4710.767875] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4721.494083] Lustre: lustre-MDD0000: changelog on [ 4724.289847] Lustre: lustre-MDD0001: changelog on [ 6945.882621] Lustre: lustre-MDD0001: changelog off [ 6948.511881] Lustre: 136089:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6949.713547] Lustre: lustre-MDD0000: changelog off [ 6959.635821] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6959.653144] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6959.663365] Lustre: Skipped 1 previous similar message [ 6959.770264] Lustre: server umount lustre-OST0000 complete [ 6969.337123] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6969.362566] LustreError: Skipped 1 previous similar message [ 6969.369953] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6969.383077] Lustre: Skipped 1 previous similar message [ 6969.390926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6971.697273] Lustre: server umount lustre-MDT0000 complete [ 6975.680673] LustreError: 136076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789130446 with bad export cookie 12952620885729558135 [ 6975.690415] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6975.701386] LustreError: 136076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6976.067087] Lustre: server umount lustre-MDT0001 complete [ 6989.637664] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 6993.331523] Key type lgssc unregistered [ 6993.702957] LNet: 147242:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6993.712255] LNetError: 147242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6993.761657] LNet: Removed LNI 192.168.204.146@tcp [ 6995.067176] Key type .llcrypt unregistered [ 6995.071698] Key type ._llcrypt unregistered [ 7021.542783] Key type ._llcrypt registered [ 7021.545032] Key type .llcrypt registered [ 7021.691322] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 7041.073538] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 7042.346810] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7042.399909] alg: No test for adler32 (adler32-zlib) [ 7043.497693] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 7043.821791] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 7045.536265] Key type lgssc registered [ 7046.607528] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7087.501443] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7087.545849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7088.810477] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7088.831164] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7088.895888] Lustre: lustre-MDT0000: new disk, initializing [ 7088.960904] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7088.974690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7094.636667] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7109.219029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7109.359496] Lustre: 150446:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7109.416576] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7109.420411] Lustre: Skipped 1 previous similar message [ 7109.517375] Lustre: lustre-MDT0001: new disk, initializing [ 7109.667923] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7109.711423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7109.739840] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7116.561536] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7127.729752] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7134.606461] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7143.348475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7143.706485] Lustre: lustre-OST0000: new disk, initializing [ 7143.715697] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7143.726409] Lustre: 152418:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7143.879742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7151.753815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7151.768252] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7151.964800] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7152.092087] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7163.046289] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7169.662604] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_module obdecho/obdecho [ 7171.581159] LustreError: 153422:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 7171.590831] LustreError: 153422:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 7173.466690] LustreError: 153518:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 7186.402641] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7186.412298] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7186.428524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7187.944824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7187.945706] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7187.950260] Lustre: Skipped 1 previous similar message [ 7187.959920] Lustre: Skipped 1 previous similar message [ 7191.855752] Lustre: server umount lustre-MDT0000 complete [ 7196.332469] LustreError: 151244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789130667 with bad export cookie 9535656900743176041 [ 7196.341445] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7196.344700] LustreError: 151244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7196.721743] Lustre: server umount lustre-MDT0001 complete [ 7210.574663] Lustre: server umount lustre-OST0000 complete [ 7233.559886] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 7244.691213] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 7284.518721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7284.808333] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7284.872517] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7284.991613] Lustre: lustre-MDT0000: new disk, initializing [ 7285.117065] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7285.164958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7290.663855] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7303.022262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7303.094884] Lustre: 157234:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7303.147678] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7303.152377] Lustre: Skipped 1 previous similar message [ 7303.238647] Lustre: lustre-MDT0001: new disk, initializing [ 7303.379204] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7303.430873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7303.449419] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7308.446409] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7319.938542] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7326.621455] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7334.826421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7335.280619] Lustre: lustre-OST0000: new disk, initializing [ 7335.285198] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7335.294912] Lustre: 159207:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7335.405231] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7342.164786] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7342.183578] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7342.315940] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7342.882837] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7353.591067] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7358.979672] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7359.217844] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7364.067389] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7364.324923] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7368.161310] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7368.177669] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7368.193750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7368.211375] Lustre: Skipped 1 previous similar message [ 7368.234100] Lustre: Skipped 1 previous similar message [ 7372.921285] Lustre: server umount lustre-OST0000 complete [ 7378.404320] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7378.408021] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7378.424352] Lustre: Skipped 1 previous similar message [ 7378.438560] Lustre: Skipped 1 previous similar message [ 7383.524624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7383.538481] Lustre: Skipped 1 previous similar message [ 7383.944948] Lustre: server umount lustre-MDT0000 complete [ 7388.026578] LustreError: 158032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789130859 with bad export cookie 9535656900743176993 [ 7388.041322] LustreError: 158032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7388.048818] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7388.363959] Lustre: server umount lustre-MDT0001 complete [ 7393.011500] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 08:47:41 (1789130861) [ 7394.982765] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7396.884455] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 08:47:45 (1789130865) [ 7405.823103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7406.323318] LustreError: 161619:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7406.340989] LustreError: 161619:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 7406.403197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7411.237900] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7411.683093] LustreError: 161620:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7416.805628] LustreError: 161619:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7421.133548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7428.450249] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7436.678954] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7443.417366] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7451.015896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7451.385356] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7451.390463] Lustre: Skipped 1 previous similar message [ 7459.115765] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7467.425900] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7477.219437] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7477.237855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7477.246342] Lustre: Skipped 2 previous similar messages [ 7477.262148] Lustre: Skipped 2 previous similar messages [ 7480.088815] Lustre: server umount lustre-MDT0000 complete [ 7482.358326] LustreError: 162546:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7482.386347] LustreError: 162546:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7484.198922] LustreError: 161600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789130955 with bad export cookie 9535656900743177882 [ 7484.203384] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7484.220329] LustreError: 161600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7487.460584] LustreError: 161614:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7487.461799] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7487.475318] LustreError: 161614:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7490.779174] Lustre: server umount lustre-MDT0001 complete [ 7499.235581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7514.095772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7514.460516] LustreError: 165462:0:(llog.c:1655:llog_backup()) MGC192.168.204.146@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7514.466772] Lustre: 165462:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.146@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7518.479692] LustreError: 165467:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7518.503546] LustreError: 165467:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7534.562702] LustreError: 165468:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7534.592553] LustreError: 165468:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 9 previous similar messages [ 7569.610790] LustreError: 165468:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7569.629969] LustreError: 165468:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 25 previous similar messages [ 7575.520454] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7575.529080] Lustre: 165462:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.146@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7616.508418] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7621.610045] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7621.804437] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7631.744709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7635.147074] LustreError: 165469:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7635.167812] LustreError: 165469:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 44 previous similar messages [ 7646.186089] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7646.201425] LustreError: Skipped 1 previous similar message [ 7655.392752] LustreError: 148667:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000029837712 x1876039317986176/t0(0) o250->MGC192.168.204.146@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7661.073253] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7661.440886] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7669.715741] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7671.623275] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7677.858323] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7680.033778] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7701.566804] Lustre: server umount lustre-MDT0000 complete [ 7701.986444] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7701.997576] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7702.013243] Lustre: Skipped 2 previous similar messages [ 7706.265110] LustreError: 161599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789131177 with bad export cookie 9535656900743184371 [ 7706.281459] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7706.286149] LustreError: 161599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7706.708099] Lustre: server umount lustre-MDT0001 complete [ 7721.633345] Lustre: server umount lustre-OST0000 complete [ 7744.636880] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 7755.235121] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 7796.588827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7796.857436] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7796.883544] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7796.986448] Lustre: lustre-MDT0000: new disk, initializing [ 7797.078635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7797.088743] Lustre: Skipped 1 previous similar message [ 7797.101667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7802.483636] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7813.623583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7813.769386] Lustre: 170571:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7814.054457] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7818.980617] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7829.970108] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7836.550749] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7843.249247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7843.608090] Lustre: lustre-OST0000: new disk, initializing [ 7843.614932] Lustre: Skipped 1 previous similar message [ 7843.622147] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7843.627085] Lustre: Skipped 2 previous similar messages [ 7843.634468] Lustre: 172541:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7850.745910] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7851.623706] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7851.642416] Lustre: Skipped 1 previous similar message [ 7851.649181] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7851.755408] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7859.931259] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7865.429356] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7865.737250] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7870.424091] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7870.702666] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7877.611901] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7877.620023] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7877.624600] LustreError: Skipped 1 previous similar message [ 7877.626559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7877.657549] Lustre: Skipped 1 previous similar message [ 7877.669429] Lustre: Skipped 5 previous similar messages [ 7887.329491] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7887.549498] Lustre: server umount lustre-OST0000 complete [ 7897.680970] Lustre: server umount lustre-MDT0000 complete [ 7898.084354] LustreError: 172190:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7898.105896] LustreError: 172190:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 23 previous similar messages [ 7902.551859] LustreError: 170564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789131373 with bad export cookie 9535656900743201171 [ 7902.554064] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7902.565788] LustreError: 170564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7903.021742] Lustre: server umount lustre-MDT0001 complete [ 7913.048041] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 08:56:21 (1789131381) [ 7920.469633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7926.226777] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7936.070819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7941.188957] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7951.203231] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7958.505678] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7966.895437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7975.844778] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7984.167372] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8005.948407] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 08:57:54 (1789131474) [ 8012.773911] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8012.795565] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8012.813882] Lustre: Skipped 2 previous similar messages [ 8012.823234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8012.833578] Lustre: Skipped 4 previous similar messages [ 8017.182302] Lustre: server umount lustre-OST0000 complete [ 8036.320350] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8036.852243] Lustre: server umount lustre-MDT0000 complete [ 8041.007109] LustreError: 174934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789131511 with bad export cookie 9535656900743202046 [ 8041.008296] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8041.015145] LustreError: 174934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8041.456403] Lustre: server umount lustre-MDT0001 complete [ 8052.803273] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 8055.755536] Key type lgssc unregistered [ 8056.176237] LNet: 178713:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8056.186759] LNetError: 178713:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8056.220155] LNet: Removed LNI 192.168.204.146@tcp [ 8057.323061] Key type .llcrypt unregistered [ 8057.326992] Key type ._llcrypt unregistered [ 8079.401274] Key type ._llcrypt registered [ 8079.404465] Key type .llcrypt registered [ 8079.554296] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 8081.134186] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8081.212672] alg: No test for adler32 (adler32-zlib) [ 8082.394259] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 8082.683886] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 8084.376186] Key type lgssc registered [ 8085.810569] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8094.463459] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8094.479828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8110.117591] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8115.233237] LustreError: 179956:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.146@tcp: failed processing log, type 4: rc = -110 [ 8120.416967] LustreError: 179956:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8120.433937] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 8120.438532] LustreError: MGC192.168.204.146@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 8120.450254] LustreError: 179956:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 8120.456414] LustreError: 179956:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 8120.464091] LustreError: 179956:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 8120.597360] Lustre: server umount lustre-OST0000 complete [ 8120.599534] LustreError: 179956:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8129.550408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8131.084869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8135.972132] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8141.284418] LustreError: 180455:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8141.324189] LustreError: 180455:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 8145.029706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8145.349324] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8150.301739] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8158.380338] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8164.757506] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8171.269569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8171.437393] LustreError: 182274:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8171.448670] Lustre: 182274:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.146@tcp: invalid local config log lustre-OST0000: rc = -22 [ 8171.637443] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8177.843710] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8185.236435] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8190.567368] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8190.575095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8190.590133] Lustre: Skipped 1 previous similar message [ 8190.610616] Lustre: Skipped 2 previous similar messages [ 8193.758108] Lustre: server umount lustre-MDT0000 complete [ 8196.069267] LustreError: 180456:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8196.101479] LustreError: 180456:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 8197.176561] LustreError: 182303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789131668 with bad export cookie 10167613336294955213 [ 8197.178049] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8197.187593] LustreError: 182303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8198.880419] LustreError: 181185:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000035eff010 x1876040407347328/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8199.071911] Lustre: server umount lustre-MDT0001 complete [ 8213.531335] Lustre: server umount lustre-OST0000 complete [ 8222.835561] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 8235.466264] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 8278.030339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8278.365745] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8278.435960] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8278.601327] Lustre: lustre-MDT0000: new disk, initializing [ 8278.716553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8278.761707] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8284.261530] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8295.314852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8295.515412] Lustre: 185817:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8295.580549] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8295.587480] Lustre: Skipped 1 previous similar message [ 8295.702190] Lustre: lustre-MDT0001: new disk, initializing [ 8295.884181] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8295.932892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8295.945905] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8301.402265] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8312.084951] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8319.013169] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8325.876547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8326.069238] Lustre: lustre-OST0000: new disk, initializing [ 8326.072319] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8326.077871] Lustre: 187785:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8326.182369] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8333.437549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8333.454452] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8333.685196] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8333.746823] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8345.229657] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8352.013404] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8352.374166] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8360.559606] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8360.936107] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8364.516333] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8364.533905] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8364.560659] Lustre: Skipped 1 previous similar message [ 8364.574520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8364.579988] Lustre: Skipped 1 previous similar message [ 8369.653849] Lustre: server umount lustre-OST0000 complete [ 8374.754477] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8374.758284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8374.776317] Lustre: Skipped 2 previous similar messages [ 8374.791154] Lustre: Skipped 1 previous similar message [ 8379.877927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8379.900657] Lustre: Skipped 1 previous similar message [ 8381.274899] Lustre: server umount lustre-MDT0000 complete [ 8384.997135] LustreError: 187186:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8385.018225] LustreError: 187186:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8386.713888] LustreError: 185808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789131857 with bad export cookie 10167613336294955934 [ 8386.715225] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8386.725933] LustreError: 185808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8387.330589] Lustre: server umount lustre-MDT0001 complete [ 8400.552463] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 09:04:29 (1789131869) [ 8421.312848] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 8424.600890] Key type lgssc unregistered [ 8425.052151] LNet: 190735:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8425.056967] LNetError: 190735:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8425.076613] LNet: Removed LNI 192.168.204.146@tcp [ 8425.945895] Key type .llcrypt unregistered [ 8425.949451] Key type ._llcrypt unregistered [ 8442.755764] Key type ._llcrypt registered [ 8442.757092] Key type .llcrypt registered [ 8442.950811] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 8444.346931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8444.420512] alg: No test for adler32 (adler32-zlib) [ 8445.584697] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 8445.841521] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 8447.568307] Key type lgssc registered [ 8448.751164] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8457.429431] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8457.455736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8459.001256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8463.148188] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8468.455551] LustreError: 191953:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8468.471107] LustreError: 191953:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 8472.716123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8473.081058] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8478.274330] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8486.127943] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8492.825394] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8501.407331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8501.638784] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8501.642398] LustreError: 193817:0:(llog.c:1696:llog_backup()) MGC192.168.204.146@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8501.652485] Lustre: 193817:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.146@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8501.790908] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8508.632328] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8517.217210] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8522.213864] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8522.221712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8522.233441] Lustre: Skipped 2 previous similar messages [ 8522.251953] Lustre: Skipped 2 previous similar messages [ 8525.962161] Lustre: server umount lustre-MDT0000 complete [ 8527.342293] LustreError: 191953:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8527.372890] LustreError: 191953:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 8530.441128] LustreError: 191935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789132001 with bad export cookie 15974702086335344761 [ 8530.446262] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8530.459387] LustreError: 191935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8530.759313] Lustre: server umount lustre-MDT0001 complete [ 8544.498341] Lustre: server umount lustre-OST0000 complete [ 8555.604677] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 8569.827292] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 8613.026931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8613.269782] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8613.299858] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8613.418115] Lustre: lustre-MDT0000: new disk, initializing [ 8613.556840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8613.593984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8618.566647] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8629.141603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8629.310562] Lustre: 197364:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8629.375736] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8629.379472] Lustre: Skipped 1 previous similar message [ 8629.382373] LustreError: 198147:0:(llog.c:1696:llog_backup()) MGC192.168.204.146@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8629.391894] LustreError: 198147:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8629.398267] Lustre: 198147:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.146@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8629.413620] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8629.418240] Lustre: Skipped 1 previous similar message [ 8629.488384] Lustre: lustre-MDT0001: new disk, initializing [ 8629.568370] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8629.593383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8629.603753] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8634.002698] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8643.011680] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8648.290248] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8654.941965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8655.233553] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8655.236286] Lustre: Skipped 1 previous similar message [ 8655.241222] LustreError: 199306:0:(llog.c:1696:llog_backup()) MGC192.168.204.146@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8655.254176] LustreError: 199306:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8655.268802] Lustre: 199306:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.146@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8655.296802] Lustre: 199306:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8655.336711] Lustre: lustre-OST0000: new disk, initializing [ 8655.343737] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8655.353344] Lustre: 199330:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8655.457590] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8661.558438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8661.573038] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8661.711767] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8661.985591] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8672.236981] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8677.767192] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8678.101080] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8683.806483] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8684.158375] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8687.595714] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8687.597239] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8687.597816] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8687.604871] LustreError: Skipped 1 previous similar message [ 8687.614825] Lustre: Skipped 1 previous similar message [ 8692.706687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8692.719679] Lustre: Skipped 1 previous similar message [ 8697.826558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8697.838275] Lustre: Skipped 1 previous similar message [ 8700.384140] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8700.548652] Lustre: server umount lustre-OST0000 complete [ 8706.529930] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8706.544616] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8706.566817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8706.573255] Lustre: Skipped 1 previous similar message [ 8718.305996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8718.312602] Lustre: Skipped 5 previous similar messages [ 8719.328394] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8719.540009] Lustre: server umount lustre-MDT0000 complete [ 8723.429562] LustreError: 198498:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8723.465550] LustreError: 198498:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8724.593778] LustreError: 197355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789132195 with bad export cookie 15974702086335345482 [ 8724.616719] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8724.618041] LustreError: 197355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8725.193031] Lustre: server umount lustre-MDT0001 complete [ 8736.342462] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 09:10:04 (1789132204) [ 8754.196274] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 8765.341885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8765.783330] LustreError: 202460:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8765.841350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8769.864984] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8771.042733] LustreError: 202461:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8778.890742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8784.240254] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8787.270846] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8794.555517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8801.539210] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8810.678963] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8810.867458] Lustre: lustre-OST0001: new disk, initializing [ 8810.873065] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8810.879831] Lustre: 204770:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8811.010106] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8811.024646] Lustre: Skipped 2 previous similar messages [ 8817.886244] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8818.259649] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8818.270908] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8818.372225] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8829.466947] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8838.945722] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8841.303072] Lustre: DEBUG MARKER: ADD OST3 [ 8846.999894] loop: module loaded [ 8847.117184] loop0: detected capacity change from 0 to 204800000 [ 8847.334390] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8847.342626] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8847.361734] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8854.936211] Lustre: DEBUG MARKER: STOP OST3 [ 8855.127384] loop0: detected capacity change from 0 to 204800000 [ 8861.547164] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8861.841443] Lustre: lustre-OST0003: new disk, initializing [ 8861.847354] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8861.871704] Lustre: 206809:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8863.408353] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8863.554605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8863.572801] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8863.658292] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8865.440666] LustreError: 206935:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8865.579885] Lustre: Failing over lustre-OST0003 [ 8865.726364] Lustre: server umount lustre-OST0003 complete [ 8868.836638] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8868.841228] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8868.852913] LustreError: 205149:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8868.860197] Lustre: Skipped 2 previous similar messages [ 8868.898556] LustreError: 205149:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 8869.711740] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8878.968183] Lustre: DEBUG MARKER: START OST3 again [ 8886.461742] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8886.859572] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8886.874080] Lustre: Skipped 1 previous similar message [ 8886.900576] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8887.503554] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8888.086929] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8888.100835] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8892.595754] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8913.670642] Lustre: Failing over lustre-OST0003 [ 8913.822995] Lustre: server umount lustre-OST0003 complete [ 8913.890575] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8913.895272] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8913.902083] LustreError: Skipped 1 previous similar message [ 8913.920882] Lustre: Skipped 1 previous similar message [ 8913.921697] LustreError: 203948:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8913.945402] LustreError: 203948:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 11 previous similar messages [ 8932.839563] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8932.842094] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8932.846903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8932.893166] Lustre: Skipped 1 previous similar message [ 8938.150635] Lustre: server umount lustre-MDT0000 complete [ 8942.374181] LustreError: 202440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789132413 with bad export cookie 15974702086335346364 [ 8942.399358] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8942.402689] LustreError: 202440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8947.105907] LustreError: 202456:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8947.122722] LustreError: 202456:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 15 previous similar messages [ 8948.882247] Lustre: server umount lustre-MDT0001 complete [ 8952.959318] Lustre: server umount lustre-OST0000 complete [ 8956.597075] Lustre: server umount lustre-OST0001 complete [ 8963.888455] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 8972.879580] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 9010.978757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9011.258503] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9011.293608] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9011.344451] Lustre: lustre-MDT0000: new disk, initializing [ 9011.443244] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9016.119255] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9027.947246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9028.131338] Lustre: 212231:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9028.258954] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9028.268065] Lustre: Skipped 1 previous similar message [ 9028.442546] Lustre: lustre-MDT0001: new disk, initializing [ 9028.654941] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9028.667142] Lustre: Skipped 1 previous similar message [ 9028.751204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9028.770147] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9033.985673] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9044.624577] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9052.511379] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9060.790451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9061.253377] Lustre: lustre-OST0000: new disk, initializing [ 9061.259447] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9061.274415] Lustre: 214202:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9062.820187] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9062.839055] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9063.008598] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9067.494661] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9076.301335] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9080.925772] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9081.214325] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9085.407136] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9085.631489] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9088.482090] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9088.488103] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9088.495923] Lustre: Skipped 4 previous similar messages [ 9088.500271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9088.509709] Lustre: Skipped 8 previous similar messages [ 9093.828687] Lustre: server umount lustre-OST0000 complete [ 9103.481213] Lustre: server umount lustre-MDT0000 complete [ 9103.841301] LustreError: 212243:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9103.852465] LustreError: 212243:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9107.474957] LustreError: 212222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789132578 with bad export cookie 15974702086335348660 [ 9107.482633] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9107.493630] LustreError: 212222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9107.739698] Lustre: server umount lustre-MDT0001 complete [ 9111.376835] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 09:16:19 (1789132579) [ 9128.928903] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 9139.501433] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 9177.165166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9177.420523] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9177.462325] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9177.554524] Lustre: lustre-MDT0000: new disk, initializing [ 9177.687341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9182.140480] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9192.876753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9192.980890] Lustre: 218810:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9193.253359] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9198.028208] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9208.549550] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9214.428740] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9221.639776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9221.926913] Lustre: 220782:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9223.864392] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9227.784594] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9237.040282] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9242.093807] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9242.282724] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9246.655965] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9246.892890] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9249.249653] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9249.260919] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9249.279581] Lustre: Skipped 3 previous similar messages [ 9249.289951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9249.302082] Lustre: Skipped 4 previous similar messages [ 9254.950105] Lustre: server umount lustre-OST0000 complete [ 9273.312756] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9273.591953] Lustre: server umount lustre-MDT0000 complete [ 9274.339633] LustreError: 218818:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9274.365205] LustreError: 218818:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9277.345975] LustreError: 219605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789132748 with bad export cookie 15974702086335349542 [ 9277.355728] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9277.760351] Lustre: server umount lustre-MDT0001 complete [ 9287.260716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9287.673951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9287.680256] Lustre: Skipped 4 previous similar messages [ 9293.319169] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9302.298500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9307.247068] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9315.527689] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9321.099862] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9329.182387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9336.238993] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9344.539456] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9357.282358] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9357.303794] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9357.312124] LustreError: Skipped 1 previous similar message [ 9357.349101] Lustre: Skipped 4 previous similar messages [ 9362.077399] Lustre: server umount lustre-OST0000 complete [ 9372.906636] Lustre: server umount lustre-MDT0000 complete [ 9376.879948] LustreError: 222982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789132847 with bad export cookie 15974702086335350424 [ 9376.886909] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9376.905799] LustreError: 222982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 9377.331076] Lustre: server umount lustre-MDT0001 complete [ 9483.700545] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 9485.849473] Key type lgssc unregistered [ 9486.102627] LNet: 226670:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9486.108096] LNetError: 226670:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9486.126136] LNet: Removed LNI 192.168.204.146@tcp [ 9486.865168] Key type .llcrypt unregistered [ 9486.870746] Key type ._llcrypt unregistered [ 9515.215294] Key type ._llcrypt registered [ 9515.217292] Key type .llcrypt registered [ 9515.296238] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 9530.525960] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 9531.470824] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9531.595358] alg: No test for adler32 (adler32-zlib) [ 9532.693860] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [ 9533.007866] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 9534.696182] Key type lgssc registered [ 9535.847835] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9568.033401] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9568.058304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9569.298589] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9569.411330] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9569.533183] Lustre: lustre-MDT0000: new disk, initializing [ 9569.695672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9569.712771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9574.630230] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9584.842806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9584.956755] Lustre: 230025:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9585.026104] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9585.029933] Lustre: Skipped 1 previous similar message [ 9585.088552] Lustre: lustre-MDT0001: new disk, initializing [ 9585.222569] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9585.284642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9585.295329] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9589.515873] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9599.071356] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9604.414771] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9611.110546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9611.383619] Lustre: lustre-OST0000: new disk, initializing [ 9611.386978] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9611.393346] Lustre: 231990:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9611.460748] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9618.035512] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9621.550884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9621.569535] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9621.704287] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9629.977552] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9637.503851] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9637.715105] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9642.804852] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9643.081399] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9647.585240] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9647.588232] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9647.603347] LustreError: Skipped 1 previous similar message [ 9647.606970] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9647.631476] Lustre: Skipped 1 previous similar message [ 9647.651474] Lustre: Skipped 1 previous similar message [ 9651.556992] Lustre: server umount lustre-OST0000 complete [ 9656.805754] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9656.822464] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9656.842179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9657.825594] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9657.850509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9657.853801] Lustre: Skipped 1 previous similar message [ 9661.995680] Lustre: server umount lustre-MDT0000 complete [ 9665.830990] LustreError: 232000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789133136 with bad export cookie 327516395922312950 [ 9665.833151] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9665.844297] LustreError: 232000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9666.204519] Lustre: server umount lustre-MDT0001 complete [ 9686.686982] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 9700.760884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9701.091773] LustreError: 234925:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9701.105484] LustreError: 234925:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 9701.180714] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9706.465467] LustreError: 234926:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9706.855853] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9711.590638] LustreError: 234925:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9716.707032] LustreError: 234926:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9718.556059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9718.917783] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9724.397293] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9727.799554] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9736.112437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9736.668798] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9743.837602] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9753.717575] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9753.963894] Lustre: lustre-OST0001: new disk, initializing [ 9753.971610] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9753.984996] Lustre: 237235:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9754.140221] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9761.271568] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9762.497227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9762.514838] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9762.605424] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9773.012880] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9785.528742] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9800.683929] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 09:27:48 (1789133268) [ 9803.759248] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9803.760867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9803.791067] Lustre: Skipped 3 previous similar messages [ 9803.805965] Lustre: Skipped 2 previous similar messages [ 9808.878717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9808.894771] Lustre: Skipped 3 previous similar messages [ 9810.171643] Lustre: server umount lustre-MDT0000 complete [ 9813.988681] LustreError: 234920:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9814.007271] LustreError: 234920:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 9814.610267] LustreError: 234907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789133285 with bad export cookie 327516395922313839 [ 9814.623274] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9814.625443] LustreError: 234907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9815.188922] Lustre: server umount lustre-MDT0001 complete [ 9829.990848] Lustre: server umount lustre-OST0000 complete [ 9844.529410] Lustre: server umount lustre-OST0001 complete [ 9851.999160] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 9863.818447] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 9903.297937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9903.598787] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9903.647287] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9903.749718] Lustre: lustre-MDT0000: new disk, initializing [ 9903.832800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9903.858900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9908.474817] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9919.362893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9919.474792] Lustre: 241181:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9919.580142] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9919.586648] Lustre: Skipped 1 previous similar message [ 9919.677950] Lustre: lustre-MDT0001: new disk, initializing [ 9919.819883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9919.834785] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9924.884520] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9933.823202] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9939.271597] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9946.464501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9946.689417] Lustre: lustre-OST0000: new disk, initializing [ 9946.692497] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9946.698218] Lustre: 243148:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9948.806362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9948.816361] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9948.982301] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9953.138950] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9965.613620] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9971.834551] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9972.088319] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9978.187633] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9978.544466] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9985.008837] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9985.032126] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9985.040898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9985.049284] Lustre: Skipped 1 previous similar message [ 9985.057398] Lustre: Skipped 1 previous similar message [ 9995.232596] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9995.610527] Lustre: server umount lustre-OST0000 complete [10000.356960] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10000.385565] Lustre: Skipped 1 previous similar message [10005.473269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10005.490901] Lustre: Skipped 7 previous similar messages [10006.616814] Lustre: server umount lustre-MDT0000 complete [10010.597241] LustreError: 241189:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10010.617198] LustreError: 241189:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [10011.366643] LustreError: 241174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789133482 with bad export cookie 327516395922315974 [10011.370241] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10011.382965] LustreError: 241174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10011.812691] Lustre: server umount lustre-MDT0001 complete [10022.823024] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10023.437754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10023.450912] Lustre: Skipped 2 previous similar messages [10028.513774] LustreError: 245369:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10028.556263] LustreError: 245369:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [10029.250507] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10042.291188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10048.980777] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10058.527572] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10064.931265] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10071.914778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10078.788896] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10086.687931] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10236.386199] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10236.392548] LustreError: Skipped 1 previous similar message [10236.402843] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10236.436642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10239.016721] Lustre: server umount lustre-OST0000 complete [10247.653957] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10247.662879] LustreError: Skipped 1 previous similar message [10257.378558] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10257.805705] Lustre: server umount lustre-MDT0000 complete [10257.892302] LustreError: 245368:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10257.929497] LustreError: 245368:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [10262.422267] LustreError: 245349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789133733 with bad export cookie 327516395922316856 [10262.433892] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10262.445260] LustreError: 245349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10263.452910] Lustre: server umount lustre-MDT0001 complete [10358.606143] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [10362.028855] Key type lgssc unregistered [10362.464303] LNet: 248985:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10362.483757] LNetError: 248985:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10362.509817] LNet: Removed LNI 192.168.204.146@tcp [10363.647345] Key type .llcrypt unregistered [10363.651371] Key type ._llcrypt unregistered [10394.769432] Key type ._llcrypt registered [10394.773106] Key type .llcrypt registered [10395.021702] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [10413.582755] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [10415.136708] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10415.382643] alg: No test for adler32 (adler32-zlib) [10416.522496] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [10416.720793] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [10418.464904] Key type lgssc registered [10419.856751] Lustre: Echo OBD driver; http://www.lustre.org/ [10475.210577] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [10488.714053] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10488.749373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10490.169985] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10490.213457] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10490.311809] Lustre: lustre-MDT0000: new disk, initializing [10490.427348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10490.452852] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10497.045649] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10513.729663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10514.016306] Lustre: 253057:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10514.202774] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10514.216134] Lustre: Skipped 1 previous similar message [10514.423476] Lustre: lustre-MDT0001: new disk, initializing [10514.556815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10514.587262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10514.602866] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10521.642431] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10527.691760] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10536.241598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10536.540184] Lustre: lustre-OST0000: new disk, initializing [10536.545805] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10536.559275] Lustre: 254694:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10536.755570] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10536.959211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10536.988807] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10537.172428] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10545.307899] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10559.382324] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10559.730824] Lustre: lustre-OST0001: new disk, initializing [10559.739167] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10559.756299] Lustre: 255565:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10560.001711] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10568.360797] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10569.405855] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10569.424422] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10569.602723] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10583.618712] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10592.344294] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10610.660885] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10610.692884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10610.699603] Lustre: Skipped 2 previous similar messages [10610.717540] Lustre: Skipped 3 previous similar messages [10614.757298] Lustre: server umount lustre-MDT0000 complete [10619.244839] LustreError: 253855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789134090 with bad export cookie 10352699965250571307 [10619.246942] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10619.255526] LustreError: 253855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10619.763284] Lustre: server umount lustre-MDT0001 complete [10635.244837] Lustre: server umount lustre-OST0000 complete [10636.257076] Lustre: 250565:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789134091/real 1789134091] req@ffff8c4ccc3403c0 x1876042854781952/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789134107 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10636.302485] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10636.327927] Lustre: Skipped 1 previous similar message [10640.393441] Lustre: server umount lustre-OST0001 complete [10663.945374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10677.373581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10688.529630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10699.874381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10722.559282] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [10733.980548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10734.020270] Lustre: lustre-MDT0000: reset Object Index mappings [10734.654286] LustreError: 260581:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10734.701441] LustreError: 260581:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [10734.780141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10740.193222] LustreError: 260582:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10740.816307] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10744.317514] LustreError: 260581:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10749.411595] LustreError: 260582:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10750.926977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10750.958386] Lustre: lustre-MDT0001: reset Object Index mappings [10751.582226] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10757.604191] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10761.263210] Lustre: 261724:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10769.705584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10770.649285] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10776.743469] LustreError: 262079:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10779.854333] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10781.462441] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10785.288928] LustreError: 262077:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10785.333615] LustreError: 262077:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [10793.407424] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10799.113376] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10801.770802] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10811.857900] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10816.686746] Lustre: 263598:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10823.138702] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10823.142625] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10823.152066] LustreError: 260582:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10823.161385] LustreError: 260582:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10823.211331] Lustre: server umount lustre-MDT0000 complete [10827.895800] LustreError: 260562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789134298 with bad export cookie 10352699965250578104 [10827.914377] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10827.924516] LustreError: 260562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10828.377157] Lustre: server umount lustre-MDT0001 complete [10844.181581] Lustre: server umount lustre-OST0000 complete [10845.152451] Lustre: 250564:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789134300/real 1789134300] req@ffff8c4cc5e583c0 x1876042854836224/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789134316 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10845.224783] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10845.250906] Lustre: Skipped 3 previous similar messages [10848.928180] Lustre: 250563:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789134303/real 1789134303] req@ffff8c4ccd4d83c0 x1876042854836480/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789134319 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10849.526517] Lustre: server umount lustre-OST0001 complete [10876.507326] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 09:45:44 (1789134344) [10900.057938] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [10914.461750] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [10958.213303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10958.509558] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10958.545927] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10958.659154] Lustre: lustre-MDT0000: new disk, initializing [10958.752607] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10958.757360] Lustre: Skipped 1 previous similar message [10958.772297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10963.445696] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10976.291559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10976.532386] Lustre: 267857:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10976.604275] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10976.609027] Lustre: Skipped 1 previous similar message [10976.675902] Lustre: lustre-MDT0001: new disk, initializing [10976.820973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10976.831392] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10981.547061] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10992.443587] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10999.621032] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11008.771667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11009.341760] Lustre: lustre-OST0000: new disk, initializing [11009.353212] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11009.363834] Lustre: 269822:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11011.394534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11011.404366] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11011.583897] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11019.108907] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11031.030935] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11036.943280] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11037.193065] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11042.101731] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11042.296843] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11047.395379] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11047.404135] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11047.420556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11047.423395] Lustre: Skipped 1 previous similar message [11047.431305] Lustre: Skipped 1 previous similar message [11050.789093] Lustre: server umount lustre-OST0000 complete [11057.634457] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11057.654984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11057.663451] Lustre: Skipped 1 previous similar message [11062.563630] Lustre: server umount lustre-MDT0000 complete [11062.768117] LustreError: 267866:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11062.811280] LustreError: 267866:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [11067.272025] LustreError: 267845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789134538 with bad export cookie 10352699965250580197 [11067.296791] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11067.309784] LustreError: 267845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11067.802534] Lustre: server umount lustre-MDT0001 complete [11091.706854] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [11105.035071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11105.616320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11105.624281] Lustre: Skipped 2 previous similar messages [11111.278752] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11125.731691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11132.164027] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11136.004972] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11144.337182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11152.442083] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11162.768753] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11163.015911] Lustre: lustre-OST0001: new disk, initializing [11163.023293] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11163.041725] Lustre: 275072:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11165.038886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11165.057397] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11165.224898] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11170.700909] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11182.555568] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11186.908617] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11193.380921] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [11193.505825] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [11203.052682] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11203.070953] LustreError: Skipped 1 previous similar message [11203.084694] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11203.112980] Lustre: Skipped 1 previous similar message [11203.124771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11207.920130] Lustre: server umount lustre-MDT0000 complete [11211.254465] LustreError: 272761:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11211.291989] LustreError: 272761:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [11212.345492] LustreError: 272744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789134683 with bad export cookie 10352699965250581079 [11212.354199] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11212.357741] LustreError: 272744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11212.813886] Lustre: server umount lustre-MDT0001 complete [11228.799676] Lustre: server umount lustre-OST0000 complete [11232.736712] Lustre: 250564:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789134687/real 1789134687] req@ffff8c4cc89ded00 x1876042854953728/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789134703 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11233.705654] Lustre: server umount lustre-OST0001 complete [11254.286341] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [11266.887459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11267.896367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11267.907049] Lustre: Skipped 3 previous similar messages [11274.042270] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11285.367114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11291.603739] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11295.573767] Lustre: 279271:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11304.022026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11311.571415] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11312.703841] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [11323.181490] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11328.546713] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [11331.491929] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11343.781952] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11350.669407] Lustre: 281149:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11358.340730] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [11358.464967] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [11370.436472] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 09:53:58 (1789134838) [11388.390293] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11388.406826] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11388.448705] Lustre: Skipped 4 previous similar messages [11388.458067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11388.473498] Lustre: Skipped 4 previous similar messages [11394.054393] Lustre: server umount lustre-MDT0000 complete [11395.047611] LustreError: 278129:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11395.065804] LustreError: 278129:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 14 previous similar messages [11398.384150] LustreError: 278111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789134869 with bad export cookie 10352699965250587617 [11398.400458] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11400.179396] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11400.191553] Lustre: Skipped 5 previous similar messages [11405.082844] Lustre: server umount lustre-MDT0001 complete [11425.249242] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11425.680496] Lustre: server umount lustre-OST0000 complete [11432.017056] Lustre: server umount lustre-OST0001 complete [11443.873029] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [11461.133027] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [11529.538091] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [11546.229867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11546.500644] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11546.526475] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11546.606783] Lustre: lustre-MDT0000: new disk, initializing [11546.737071] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11546.742213] Lustre: Skipped 3 previous similar messages [11546.759382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11554.267516] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11569.224539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11569.511785] Lustre: 285421:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11570.109322] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11577.047211] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11583.011911] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11591.530860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11591.959619] Lustre: 287055:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11593.613966] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11601.517560] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11614.857133] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11615.285542] Lustre: 287925:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11617.226836] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11617.237833] Lustre: Skipped 1 previous similar message [11617.402129] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11623.683173] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11635.376825] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11657.909329] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 09:58:46 (1789135126) [11662.320810] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11662.336573] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11662.360293] Lustre: Skipped 5 previous similar messages [11662.387650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11662.395733] Lustre: Skipped 2 previous similar messages [11663.034237] Lustre: server umount lustre-MDT0000 complete [11663.337263] LustreError: 285431:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11663.368405] LustreError: 285431:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [11667.794385] LustreError: 287054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789135138 with bad export cookie 10352699965250595268 [11667.806371] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11667.812153] LustreError: 287054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11668.660641] Lustre: server umount lustre-MDT0001 complete [11673.125341] Lustre: server umount lustre-OST0000 complete [11678.712636] Lustre: server umount lustre-OST0001 complete [11690.939357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11697.368751] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11708.526346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11714.310938] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11723.675829] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11731.353298] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11741.279709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11750.547860] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11761.202537] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11784.737393] loop1: detected capacity change from 0 to 204800000 [11785.058764] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11785.083792] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11785.108283] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11791.686970] loop1: detected capacity change from 0 to 204800000 [11798.883220] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11799.000370] LustreError: 290160:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs7058: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11799.014387] LustreError: 290160:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs7058 [11799.019581] LustreError: 290160:0:(mgs_handler.c:517:mgs_target_reg()) fs7058-MDT0001: Error checking target index: rc = -13 [11799.027764] LustreError: fs7058-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11799.034471] LustreError: 293906:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11799.039521] LustreError: 293906:0:(tgt_mount.c:2129:server_put_super()) no obd fs7058-MDT0001 [11799.044569] LustreError: 293906:0:(tgt_mount.c:116:server_deregister_mount()) fs7058-MDT0001 not registered [11799.088210] Lustre: server umount fs7058-MDT0001 complete [11799.092943] LustreError: 293906:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11809.912687] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11810.182398] Lustre: Setting parameter fs7058-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs7058-MDT0001 [11810.199441] Lustre: Skipped 1 previous similar message [11810.334365] Lustre: srv-fs7058-MDT0001: No data found on store. Initialize space. [11810.345981] Lustre: Skipped 4 previous similar messages [11810.551265] Lustre: fs7058-MDT0001: new disk, initializing [11810.574039] Lustre: Skipped 3 previous similar messages [11810.714996] Lustre: srv-fs7058-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11810.732277] Lustre: 294372:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs7058-MDT0001-mdtlov: can't create new llog: rc = -115 [11810.751461] LustreError: 294372:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs7058-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11816.843900] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11826.615404] Lustre: server umount fs7058-MDT0001 complete [11833.325397] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11846.177055] loop1: detected capacity change from 0 to 204800000 [11853.586659] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11854.000919] Lustre: 295942:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs7058-MDT0001-mdtlov: can't create new llog: rc = -115 [11854.012962] LustreError: 295942:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs7058-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11859.353492] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11864.932644] Lustre: server umount fs7058-MDT0001 complete [11877.859598] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11877.873326] LustreError: Skipped 1 previous similar message [11877.886423] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11877.926444] Lustre: Skipped 5 previous similar messages [11877.936979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11882.501127] Lustre: server umount lustre-MDT0000 complete [11893.223776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11893.397819] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11899.147575] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11899.398267] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11908.090197] loop1: detected capacity change from 0 to 204800000 [11913.903961] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11914.337080] Lustre: 298390:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs7058-MDT0001-mdtlov: can't create new llog: rc = -115 [11914.361278] LustreError: 298390:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs7058-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11919.524132] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11924.803241] Lustre: server umount fs7058-MDT0001 complete [11950.565257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11950.584513] Lustre: Skipped 4 previous similar messages [11955.174301] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11956.851515] Lustre: server umount lustre-OST0000 complete [11978.208143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11978.722402] Lustre: server umount lustre-MDT0000 complete [11985.007323] LustreError: 290152:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789135455 with bad export cookie 10352699965250597949 [11985.012251] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11985.024833] LustreError: 290152:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11985.678399] Lustre: server umount lustre-MDT0001 complete [11999.615239] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [12002.872883] Key type lgssc unregistered [12003.336931] LNet: 300738:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12003.356184] LNetError: 300738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12003.378938] LNet: Removed LNI 192.168.204.146@tcp [12004.622461] Key type .llcrypt unregistered [12004.633356] Key type ._llcrypt unregistered [12032.425670] Key type ._llcrypt registered [12032.427220] Key type .llcrypt registered [12032.602281] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [12033.624186] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12033.645946] alg: No test for adler32 (adler32-zlib) [12034.796253] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [12035.096213] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [12036.865980] Key type lgssc registered [12038.097484] Lustre: Echo OBD driver; http://www.lustre.org/ [12047.707270] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12047.723843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12049.079722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12054.843617] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12059.625124] LustreError: 302012:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12059.662896] LustreError: 302012:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12064.751129] LustreError: 302013:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12067.895579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12068.544851] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12074.845867] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12084.690814] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12092.220576] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12103.162623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12104.131284] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12113.218223] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12113.352948] LustreError: 303846:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12118.508196] LustreError: 303847:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12122.897311] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12124.648337] LustreError: 304214:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12124.667854] LustreError: 304214:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [12134.893951] LustreError: 303845:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12134.916573] LustreError: 303845:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [12143.002923] loop1: detected capacity change from 0 to 204800000 [12143.215401] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12143.225604] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12143.246971] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12149.424359] loop1: detected capacity change from 0 to 204800000 [12151.520681] LustreError: 303847:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12151.565356] LustreError: 303847:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 9 previous similar messages [12157.407375] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12157.588777] Lustre: fs11808-OST0000: new disk, initializing [12157.592968] Lustre: srv-fs11808-OST0000: No data found on store. Initialize space. [12157.597846] Lustre: 305753:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12157.692298] Lustre: fs11808-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12166.202908] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12176.237338] Lustre: server umount fs11808-OST0000 complete [12186.090757] LustreError: 304824:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12186.120622] LustreError: 304824:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 19 previous similar messages [12191.236090] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12194.037612] loop1: detected capacity change from 0 to 204800000 [12196.343306] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12204.288726] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12204.860402] Lustre: fs11808-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12215.593580] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12227.676120] loop2: detected capacity change from 0 to 204800000 [12227.910696] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12227.922614] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12227.936590] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12235.336404] loop2: detected capacity change from 0 to 204800000 [12242.356691] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12242.450288] LustreError: fs11808-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [12242.460725] LustreError: fs11808-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [12242.471118] LustreError: 308798:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [12242.488914] LustreError: 308798:0:(tgt_mount.c:2129:server_put_super()) no obd fs11808-OST0001 [12242.497449] LustreError: 308798:0:(tgt_mount.c:116:server_deregister_mount()) fs11808-OST0001 not registered [12242.538635] Lustre: server umount fs11808-OST0001 complete [12242.543546] LustreError: 308798:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [12252.659711] LustreError: 303846:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12252.683218] LustreError: 303846:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 38 previous similar messages [12254.784070] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12254.886665] Lustre: fs11808-OST0001: new disk, initializing [12254.890145] Lustre: srv-fs11808-OST0001: No data found on store. Initialize space. [12254.896632] Lustre: 309349:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12254.969266] Lustre: fs11808-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12262.079994] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12271.170283] Lustre: server umount fs11808-OST0000 complete [12281.705425] Lustre: server umount fs11808-OST0001 complete [12297.219735] loop1: detected capacity change from 0 to 204800000 [12305.263705] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12305.678722] Lustre: fs11808-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12315.383870] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12325.041057] loop2: detected capacity change from 0 to 204800000 [12332.450647] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12340.085658] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12342.771276] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12347.887636] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12351.447697] Lustre: server umount fs11808-OST0000 complete [12361.627500] Lustre: server umount fs11808-OST0001 complete [12375.525105] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12375.543294] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12375.569684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12376.546498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12376.563450] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12381.687901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12381.701148] LustreError: 303845:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12381.712458] Lustre: Skipped 1 previous similar message [12381.720620] LustreError: 303845:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 75 previous similar messages [12386.797902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12389.345985] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12389.878194] Lustre: server umount lustre-OST0000 complete [12397.034439] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12397.043573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12397.059909] Lustre: Skipped 1 previous similar message [12397.108352] Lustre: Skipped 2 previous similar messages [12403.042301] Lustre: server umount lustre-MDT0000 complete [12407.403526] LustreError: 301993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789135878 with bad export cookie 8860965942766956366 [12407.405644] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12407.413433] LustreError: 301993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12407.929135] Lustre: server umount lustre-MDT0001 complete [12421.988879] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [12425.416621] Key type lgssc unregistered [12425.704930] LNet: 315280:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12425.722143] LNetError: 315280:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12425.743691] LNet: Removed LNI 192.168.204.146@tcp [12426.668189] Key type .llcrypt unregistered [12426.669871] Key type ._llcrypt unregistered [12464.246306] Key type ._llcrypt registered [12464.248878] Key type .llcrypt registered [12464.407942] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [12482.246370] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [12483.256960] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12483.282848] alg: No test for adler32 (adler32-zlib) [12484.362904] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [12484.644463] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [12486.336187] Key type lgssc registered [12487.609883] Lustre: Echo OBD driver; http://www.lustre.org/ [12534.916342] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12534.939424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12536.258370] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12536.316943] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12536.426958] Lustre: lustre-MDT0000: new disk, initializing [12536.565508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12536.587030] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12541.578167] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12553.080150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12553.250745] Lustre: 319143:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12553.334461] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12553.340030] Lustre: Skipped 1 previous similar message [12553.452325] Lustre: lustre-MDT0001: new disk, initializing [12553.558393] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12553.620716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12553.627888] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12558.640295] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12570.121106] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12577.368751] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12585.607157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12585.965837] Lustre: lustre-OST0000: new disk, initializing [12585.970188] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12585.978147] Lustre: 321114:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12586.075224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12592.907870] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12595.971815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12596.002547] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12596.235964] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12603.625332] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12605.744573] Lustre: Mounted lustre-client - version 2.17.57_138_g7946b16 [12606.746509] Lustre: Unmounted lustre-client [12616.675722] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12616.692944] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12616.712325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12616.722404] Lustre: Skipped 1 previous similar message [12621.283389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12621.295804] Lustre: Skipped 1 previous similar message [12622.887326] Lustre: server umount lustre-OST0000 complete [12630.501972] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12630.530527] LustreError: Skipped 1 previous similar message [12630.552704] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12630.583601] Lustre: Skipped 1 previous similar message [12630.597272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12633.783615] Lustre: server umount lustre-MDT0000 complete [12638.679894] LustreError: 319136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789136109 with bad export cookie 12549857546695615860 [12638.688845] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12638.704943] LustreError: 319136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12639.230384] Lustre: server umount lustre-MDT0001 complete [12653.602303] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [12657.012848] Key type lgssc unregistered [12667.682427] LNet: 323280:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12667.697415] LNetError: 323280:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12736.361171] LNet: Removed LNI 192.168.204.146@tcp [12737.725594] Key type .llcrypt unregistered [12737.728157] Key type ._llcrypt unregistered [12761.434585] Key type ._llcrypt registered [12761.436376] Key type .llcrypt registered [12761.593578] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [12780.074599] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [12781.112989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12781.417154] alg: No test for adler32 (adler32-zlib) [12782.462970] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [12782.681520] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [12784.376308] Key type lgssc registered [12785.562743] Lustre: Echo OBD driver; http://www.lustre.org/ [12820.518910] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 10:18:08 (1789136288) [12828.119300] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12828.168223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12829.438105] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12829.475149] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12829.607583] Lustre: lustre-MDT0000: new disk, initializing [12829.781348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12829.808302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12834.478274] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12845.537513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12845.657833] Lustre: 326506:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12845.723696] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12845.727919] Lustre: Skipped 1 previous similar message [12845.802807] Lustre: lustre-MDT0001: new disk, initializing [12845.900938] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12845.942346] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12845.959125] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12850.181466] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12860.913427] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12867.788478] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12875.921294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12876.307160] Lustre: lustre-OST0000: new disk, initializing [12876.312359] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12876.321572] Lustre: 328475:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12876.449626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12883.796103] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12885.620225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12885.634699] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12885.853389] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12894.455114] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12911.590066] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12911.604713] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12911.632888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12915.984478] Lustre: server umount lustre-OST0000 complete [12921.826479] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12921.853371] Lustre: Skipped 1 previous similar message [12921.867941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12921.878274] Lustre: Skipped 2 previous similar messages [12926.945853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12926.959051] Lustre: Skipped 1 previous similar message [12931.046465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12931.062021] Lustre: Skipped 1 previous similar message [12934.114583] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12934.345502] Lustre: server umount lustre-MDT0000 complete [12939.254253] LustreError: 326497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789136410 with bad export cookie 1715370883801919630 [12939.256069] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12939.276286] LustreError: 326497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12940.124376] Lustre: server umount lustre-MDT0001 complete [12955.155895] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [12958.072521] Key type lgssc unregistered [12958.564758] LNet: 330388:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12958.578984] LNetError: 330388:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12959.654922] LNet: Removed LNI 192.168.204.146@tcp [12960.966117] Key type .llcrypt unregistered [12960.968614] Key type ._llcrypt unregistered [12985.177469] Key type ._llcrypt registered [12985.179797] Key type .llcrypt registered [12985.321025] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [12986.797272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12986.878274] alg: No test for adler32 (adler32-zlib) [12988.104657] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [12988.414978] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [12990.281401] Key type lgssc registered [12991.516708] Lustre: Echo OBD driver; http://www.lustre.org/ [12999.722688] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12999.736765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13001.401166] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13006.665987] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13011.941374] LustreError: 331672:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13011.959442] LustreError: 331672:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [13015.572119] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13015.887772] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13020.607377] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13028.839988] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13035.914457] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13043.384047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13043.724079] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13050.425986] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13058.383916] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13070.624276] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13072.619929] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13077.476101] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13077.485970] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13077.504445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13080.546257] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13080.552568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13080.564381] Lustre: Skipped 1 previous similar message [13080.586257] Lustre: Skipped 2 previous similar messages [13081.857567] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.46@tcp (stopping) [13083.704431] LustreError: 334590:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x2:0x0].0x0 (00000000852a09cc) refcount nonzero (1) after lock cleanup; forcing cleanup. [13083.739177] Lustre: server umount lustre-MDT0000 complete [13085.668873] LustreError: 332401:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13085.702211] LustreError: 332401:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [13086.938967] LustreError: 331669:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13088.085599] LustreError: 331654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789136559 with bad export cookie 1323407555812168452 [13088.107590] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13088.125495] LustreError: 331654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13090.789851] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13090.795859] LustreError: 334511:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13090.808827] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13094.565399] Lustre: server umount lustre-MDT0001 complete [13098.999528] Lustre: server umount lustre-OST0000 complete [13109.698924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13114.701292] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13118.000329] Lustre: 335854:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.146@tcp [13120.184807] Lustre: server umount MGS complete [13128.515371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13129.096752] LustreError: 336425:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13129.120121] LustreError: 336425:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [13129.168227] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13134.053632] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13137.102448] LustreError: 336422:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13137.139058] LustreError: 336422:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [13143.112732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13143.527831] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13148.196212] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13157.726098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13158.103591] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13164.274408] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13173.813738] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13175.616039] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [13183.941837] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13185.799861] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13192.169459] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [13192.177988] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13192.200265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13192.218695] Lustre: Skipped 2 previous similar messages [13192.529911] Lustre: server umount lustre-OST0000 complete [13199.846033] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13199.858912] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13200.358129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13200.369289] Lustre: Skipped 2 previous similar messages [13203.358333] Lustre: server umount lustre-MDT0000 complete [13205.477986] LustreError: 337154:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13205.513735] LustreError: 337154:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [13207.238534] LustreError: 336408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789136678 with bad export cookie 1323407555812169292 [13207.247820] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13217.933140] Lustre: server umount lustre-MDT0001 complete [13229.036302] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [13231.639752] Key type lgssc unregistered [13231.944749] LNet: 339867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13231.955646] LNetError: 339867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13231.986161] LNet: Removed LNI 192.168.204.146@tcp [13232.711192] Key type .llcrypt unregistered [13232.713492] Key type ._llcrypt unregistered [13252.544400] Key type ._llcrypt registered [13252.546215] Key type .llcrypt registered [13252.622698] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [13268.459259] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [13269.479783] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13269.572282] alg: No test for adler32 (adler32-zlib) [13270.631472] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [13270.851560] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [13272.504488] Key type lgssc registered [13273.753762] Lustre: Echo OBD driver; http://www.lustre.org/ [13310.405186] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13310.433757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13311.674950] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13311.708734] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13311.761776] Lustre: lustre-MDT0000: new disk, initializing [13311.845295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13311.867673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13316.206875] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13329.249779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13329.492455] Lustre: 343031:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13329.629138] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13329.637782] Lustre: Skipped 1 previous similar message [13329.825802] Lustre: lustre-MDT0001: new disk, initializing [13330.091113] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13330.203852] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13330.225345] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13336.754032] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13348.099618] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13354.395822] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13361.439678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13361.754992] Lustre: lustre-OST0000: new disk, initializing [13361.759418] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13361.766064] Lustre: 344999:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13361.862074] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13369.141774] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13370.483602] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13370.500017] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13370.635678] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13379.967720] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13385.256465] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13385.482814] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13390.675700] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13390.903664] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13396.454963] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13396.466150] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13396.489219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13396.496569] Lustre: Skipped 1 previous similar message [13399.910470] Lustre: server umount lustre-OST0000 complete [13406.698945] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13406.701484] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13406.723037] LustreError: Skipped 1 previous similar message [13406.728168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13406.750650] Lustre: Skipped 2 previous similar messages [13411.162958] Lustre: server umount lustre-MDT0000 complete [13414.720775] LustreError: 343022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789136885 with bad export cookie 10134281795338975095 [13414.724100] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13414.733329] LustreError: 343022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13415.139750] Lustre: server umount lustre-MDT0001 complete [13424.153440] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 10:28:12 (1789136892) [13439.871723] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [13450.435449] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [13496.557875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13496.764402] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13496.787438] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13496.876825] Lustre: lustre-MDT0000: new disk, initializing [13496.946625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13496.961946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13502.172965] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13513.607523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13513.684486] Lustre: 349760:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13513.732541] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13513.736758] Lustre: Skipped 1 previous similar message [13513.797346] Lustre: lustre-MDT0001: new disk, initializing [13513.883367] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13513.935835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13513.950689] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13518.707730] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13529.320282] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13535.542492] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13544.213252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13544.663789] Lustre: lustre-OST0000: new disk, initializing [13544.667667] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13544.673758] Lustre: 351731:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13544.800646] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13551.218350] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13551.245035] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13551.404946] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13552.526306] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13562.044346] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13597.665372] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13597.668595] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13597.677285] LustreError: Skipped 1 previous similar message [13597.678360] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13597.678366] Lustre: Skipped 1 previous similar message [13597.704327] Lustre: Skipped 1 previous similar message [13601.649246] Lustre: server umount lustre-OST0000 complete [13606.372514] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13606.380603] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13606.392091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13606.397025] Lustre: Skipped 1 previous similar message [13612.134639] Lustre: server umount lustre-MDT0000 complete [13613.034508] LustreError: 352601:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13613.063218] LustreError: 352601:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [13615.891853] LustreError: 350556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789137086 with bad export cookie 10134281795338975977 [13615.895340] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13615.901110] LustreError: 350556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13616.197655] Lustre: server umount lustre-MDT0001 complete [13628.165469] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [13630.975673] Key type lgssc unregistered [13631.239501] LNet: 353908:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13631.246565] LNetError: 353908:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13631.261745] LNet: Removed LNI 192.168.204.146@tcp [13632.147940] Key type .llcrypt unregistered [13632.153086] Key type ._llcrypt unregistered [13729.643293] Key type ._llcrypt registered [13729.646153] Key type .llcrypt registered [13729.736743] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [13730.096302] Key type .llcrypt unregistered [13730.099970] Key type ._llcrypt unregistered [13818.591782] Key type ._llcrypt registered [13818.595252] Key type .llcrypt registered [13818.732875] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [13819.105766] Key type .llcrypt unregistered [13819.111417] Key type ._llcrypt unregistered [13852.181468] Key type ._llcrypt registered [13852.184357] Key type .llcrypt registered [13852.310236] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [13852.586572] Key type .llcrypt unregistered [13852.588774] Key type ._llcrypt unregistered [13885.496557] Key type ._llcrypt registered [13885.499454] Key type .llcrypt registered [13885.670828] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [13886.182539] Key type .llcrypt unregistered [13886.187433] Key type ._llcrypt unregistered [13980.326570] Key type ._llcrypt registered [13980.329232] Key type .llcrypt registered [13980.418531] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [13980.639332] Key type .llcrypt unregistered [13980.642361] Key type ._llcrypt unregistered [14011.931808] Key type ._llcrypt registered [14011.933792] Key type .llcrypt registered [14012.128313] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [14012.719307] Key type .llcrypt unregistered [14012.723672] Key type ._llcrypt unregistered [14051.722975] Key type ._llcrypt registered [14051.727399] Key type .llcrypt registered [14051.834270] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [14052.186670] Key type .llcrypt unregistered [14052.191715] Key type ._llcrypt unregistered [14087.433632] Key type ._llcrypt registered [14087.435341] Key type .llcrypt registered [14087.575758] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [14087.976642] Key type .llcrypt unregistered [14087.979097] Key type ._llcrypt unregistered [14123.200430] Key type ._llcrypt registered [14123.203897] Key type .llcrypt registered [14123.310755] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [14123.714667] Key type .llcrypt unregistered [14123.717387] Key type ._llcrypt unregistered [14159.425656] Key type ._llcrypt registered [14159.428911] Key type .llcrypt registered [14159.526663] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [14159.842319] Key type .llcrypt unregistered [14159.846132] Key type ._llcrypt unregistered [14235.063186] Key type ._llcrypt registered [14235.067147] Key type .llcrypt registered [14235.158932] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [14235.447884] Key type .llcrypt unregistered [14235.452139] Key type ._llcrypt unregistered [14270.288835] Key type ._llcrypt registered [14270.294072] Key type .llcrypt registered [14270.462987] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [14270.883333] Key type .llcrypt unregistered [14270.887534] Key type ._llcrypt unregistered [14303.435475] Key type ._llcrypt registered [14303.438937] Key type .llcrypt registered [14303.588568] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [14303.950383] Key type .llcrypt unregistered [14303.954497] Key type ._llcrypt unregistered [14363.772348] Key type ._llcrypt registered [14363.776585] Key type .llcrypt registered [14363.918444] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [14381.665781] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [14382.832331] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14382.968906] alg: No test for adler32 (adler32-zlib) [14384.084217] Lustre: Lustre: Build Version: 2.17.57_138_g7946b16 [14384.320351] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [14385.992256] Key type lgssc registered [14387.166371] Lustre: Echo OBD driver; http://www.lustre.org/ [14426.435120] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14426.454037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14427.659192] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [14427.689422] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [14427.797966] Lustre: tst250fs-MDT0000: new disk, initializing [14427.879783] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14427.893506] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14433.617558] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [14446.166515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14446.326577] Lustre: 374304:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14446.422457] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [14446.428217] Lustre: Skipped 1 previous similar message [14446.580631] Lustre: tst250fs-MDT0001: new disk, initializing [14446.592164] Lustre: tst250fs-MDT0001: Not available for connect from 0@lo (not set up) [14446.751675] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14446.810961] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14446.848908] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14453.025606] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [14459.277599] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [14467.071611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14467.449591] Lustre: tst250fs-OST0000: new disk, initializing [14467.455469] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [14467.464032] Lustre: 375937:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14467.584183] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14473.775571] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14473.786170] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14473.916882] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14474.713418] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [14487.002468] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14487.181857] Lustre: tst250fs-OST0001: new disk, initializing [14487.185909] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [14487.190993] Lustre: 376809:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14487.297830] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14494.036522] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [14494.815298] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [14494.826437] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14494.917515] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [14499.932600] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [14502.488902] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [14502.498887] Lustre: Skipped 2 previous similar messages [14525.937669] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14525.942268] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14525.958759] Lustre: Skipped 3 previous similar messages [14525.975550] Lustre: Skipped 2 previous similar messages [14530.934583] Lustre: server umount tst250fs-MDT0000 complete [14535.314862] LustreError: 374295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789138006 with bad export cookie 16025522451888968984 [14535.322226] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14535.329312] LustreError: 374295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14535.906994] Lustre: server umount tst250fs-MDT0001 complete [14551.022514] Lustre: server umount tst250fs-OST0000 complete [14552.868892] Lustre: 372524:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789138007/real 1789138007] req@ffff8c4cc2b9cb40 x1876047015517440/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789138023 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14552.901520] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14555.219815] Lustre: server umount tst250fs-OST0001 complete [14575.614851] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [14584.831778] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [14617.359617] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 10:48:05 (1789138085) [14619.284081] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [14638.295612] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [14651.003942] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [14690.766276] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14383 sec ========= 10:49:18 (1789138158) [14693.245518] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 10:49:21 (1789138161) === [14697.968258] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 10:49:25 (1789138165) === [14751.120577] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [14754.137458] Key type lgssc unregistered [14754.442799] LNet: 385955:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14754.465100] LNetError: 385955:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14754.479609] LNet: Removed LNI 192.168.204.146@tcp [14755.473239] Key type .llcrypt unregistered [14755.483541] Key type ._llcrypt unregistered