[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 492480686 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002336] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004011] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008011] pid_max: default: 32768 minimum: 301 [ 0.009121] LSM: Security Framework initializing [ 0.010045] Yama: becoming mindful. [ 0.011036] SELinux: Initializing. [ 0.012089] *** VALIDATE selinux *** [ 0.020705] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025799] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027091] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028106] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030059] *** VALIDATE tmpfs *** [ 0.031463] *** VALIDATE proc *** [ 0.033128] *** VALIDATE cgroup *** [ 0.034008] *** VALIDATE cgroup2 *** [ 0.035247] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036148] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038028] Spectre V2 : User space: Vulnerable [ 0.039008] Speculative Store Bypass: Vulnerable [ 0.042117] debug: unmapping init [mem 0xffffffff8dc59000-0xffffffff8dc60fff] [ 0.044747] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045628] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046022] ... version: 2 [ 0.047010] ... bit width: 48 [ 0.048014] ... generic registers: 4 [ 0.049011] ... value mask: 0000ffffffffffff [ 0.050012] ... max period: 00007fffffffffff [ 0.051013] ... fixed-purpose events: 3 [ 0.052010] ... event mask: 000000070000000f [ 0.054209] rcu: Hierarchical SRCU implementation. [ 0.056407] smp: Bringing up secondary CPUs ... [ 0.057591] x86: Booting SMP configuration: [ 0.058022] .... node #0, CPUs: #1 #2 #3 [ 0.062014] smp: Brought up 1 node, 4 CPUs [ 0.064018] smpboot: Max logical packages: 1 [ 0.065011] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.201899] node 0 deferred pages initialised in 135ms [ 0.206141] devtmpfs: initialized [ 0.207234] x86/mm: Memory block size: 128MB [ 0.209712] gcov: version magic: 0x41383552 [ 0.212319] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.213106] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.216427] pinctrl core: initialized pinctrl subsystem [ 0.219489] [ 0.220011] ************************************************************* [ 0.224024] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.227019] ** ** [ 0.230022] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.233020] ** ** [ 0.236022] ** This means that this kernel is built to expose internal ** [ 0.239031] ** IOMMU data structures, which may compromise security on ** [ 0.242020] ** your system. ** [ 0.245033] ** ** [ 0.248024] ** If you see this message and you are not debugging the ** [ 0.250018] ** kernel, report this immediately to your vendor! ** [ 0.253020] ** ** [ 0.255018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.258019] ************************************************************* [ 0.261476] NET: Registered protocol family 16 [ 0.264435] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.268063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.271072] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.276050] cpuidle: using governor menu [ 0.278973] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.281761] PCI: Using configuration type 1 for base access [ 0.284154] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.296122] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.299033] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.302051] cryptd: max_cpu_qlen set to 1000 [ 0.306177] ACPI: Added _OSI(Module Device) [ 0.308027] ACPI: Added _OSI(Processor Device) [ 0.310022] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.312031] ACPI: Added _OSI(Processor Aggregator Device) [ 0.318354] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.327354] ACPI: Interpreter enabled [ 0.329075] ACPI: PM: (supports S0 S3 S4 S5) [ 0.330038] ACPI: Using IOAPIC for interrupt routing [ 0.332132] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.336481] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.348688] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.351055] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.354019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.358080] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.364448] acpiphp: Slot [2] registered [ 0.366169] acpiphp: Slot [5] registered [ 0.368157] acpiphp: Slot [6] registered [ 0.369157] acpiphp: Slot [7] registered [ 0.371130] acpiphp: Slot [8] registered [ 0.373132] acpiphp: Slot [9] registered [ 0.374149] acpiphp: Slot [10] registered [ 0.376174] acpiphp: Slot [3] registered [ 0.378095] acpiphp: Slot [4] registered [ 0.380108] acpiphp: Slot [11] registered [ 0.382114] acpiphp: Slot [12] registered [ 0.383094] acpiphp: Slot [13] registered [ 0.385097] acpiphp: Slot [14] registered [ 0.387116] acpiphp: Slot [15] registered [ 0.388094] acpiphp: Slot [16] registered [ 0.390144] acpiphp: Slot [17] registered [ 0.392099] acpiphp: Slot [18] registered [ 0.394118] acpiphp: Slot [19] registered [ 0.395100] acpiphp: Slot [20] registered [ 0.397071] acpiphp: Slot [21] registered [ 0.399087] acpiphp: Slot [22] registered [ 0.400086] acpiphp: Slot [23] registered [ 0.402105] acpiphp: Slot [24] registered [ 0.403105] acpiphp: Slot [25] registered [ 0.405087] acpiphp: Slot [26] registered [ 0.406076] acpiphp: Slot [27] registered [ 0.407086] acpiphp: Slot [28] registered [ 0.408083] acpiphp: Slot [29] registered [ 0.410099] acpiphp: Slot [30] registered [ 0.411000] acpiphp: Slot [31] registered [ 0.411000] PCI host bridge to bus 0000:00 [ 0.413023] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.415019] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.418045] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.420021] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.423021] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.426023] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.427192] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.431046] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.434408] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.445013] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.450051] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.452016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.454013] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.458017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.459630] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.461917] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.464038] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.467052] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.472016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.486015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.492023] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.500456] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.516016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.527014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.542015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.550787] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.560014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.576020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.604015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.618059] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.627016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.638015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.658015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.671104] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.691015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.697029] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.725018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.737404] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.746015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.758017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.784016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.800497] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.806015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.813014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.834020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.852351] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.855403] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.856504] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.859350] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.861206] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.866043] iommu: Default domain type: Passthrough [ 0.867378] SCSI subsystem initialized [ 0.869123] ACPI: bus type USB registered [ 0.870107] usbcore: registered new interface driver usbfs [ 0.872064] usbcore: registered new interface driver hub [ 0.874064] usbcore: registered new device driver usb [ 0.875155] pps_core: LinuxPPS API ver. 1 registered [ 0.876010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.878048] PTP clock support registered [ 0.880125] EDAC MC: Ver: 3.0.0 [ 0.882136] PCI: Using ACPI for IRQ routing [ 0.883796] NetLabel: Initializing [ 0.886035] NetLabel: domain hash size = 128 [ 0.887015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.889094] NetLabel: unlabeled traffic allowed by default [ 0.891243] vgaarb: loaded [ 0.893295] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.894014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.901318] clocksource: Switched to clocksource kvm-clock [ 1.017664] VFS: Disk quotas dquot_6.6.0 [ 1.019483] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.022286] *** VALIDATE ramfs *** [ 1.023152] *** VALIDATE hugetlbfs *** [ 1.024495] pnp: PnP ACPI init [ 1.027176] pnp: PnP ACPI: found 6 devices [ 1.048067] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.051689] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.053951] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.056193] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.059063] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.061807] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.065041] NET: Registered protocol family 2 [ 1.067529] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.073042] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.076869] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.084589] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.088914] TCP: Hash tables configured (established 65536 bind 65536) [ 1.091172] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.094213] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.097048] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.099833] NET: Registered protocol family 1 [ 1.102386] RPC: Registered named UNIX socket transport module. [ 1.104468] RPC: Registered udp transport module. [ 1.106259] RPC: Registered tcp transport module. [ 1.107879] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.110652] NET: Registered protocol family 44 [ 1.112642] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.115178] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.117569] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.120122] PCI: CLS 0 bytes, default 64 [ 1.122927] Unpacking initramfs... [ 2.593093] debug: unmapping init [mem 0xffff90d2fcc54000-0xffff90d2fffbffff] [ 2.601027] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.602790] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.606252] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.137923] Initialise system trusted keyrings [ 3.139861] Key type blacklist registered [ 3.141349] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.149954] zbud: loaded [ 3.152764] *** VALIDATE nfs *** [ 3.154094] *** VALIDATE nfs4 *** [ 3.155852] pstore: using deflate compression [ 3.159287] Platform Keyring initialized [ 3.260153] NET: Registered protocol family 38 [ 3.261525] Key type asymmetric registered [ 3.262643] Asymmetric key parser 'x509' registered [ 3.263970] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.266399] io scheduler mq-deadline registered [ 3.267620] io scheduler kyber registered [ 3.269272] io scheduler bfq registered [ 3.270865] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.273275] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.275561] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.277873] ACPI: Power Button [PWRF] [ 3.282986] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.291334] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.310088] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.317684] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.339096] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.366960] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.393910] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.399226] Non-volatile memory driver v1.3 [ 3.400597] Linux agpgart interface v0.103 [ 3.434470] virtio_blk virtio1: [vda] 146936 512-byte logical blocks (75.2 MB/71.7 MiB) [ 3.437128] vda: detected capacity change from 0 to 75231232 [ 3.453334] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.458272] vdb: detected capacity change from 0 to 1073741824 [ 3.474854] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.478185] vdc: detected capacity change from 0 to 2621440000 [ 3.493609] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.496561] vdd: detected capacity change from 0 to 2621440000 [ 3.509934] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.512921] vde: detected capacity change from 0 to 4294967296 [ 3.526949] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.531041] vdf: detected capacity change from 0 to 4294967296 [ 3.536912] libphy: Fixed MDIO Bus: probed [ 3.541235] usbcore: registered new interface driver usbserial_generic [ 3.543274] usbserial: USB Serial support registered for generic [ 3.545136] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.548605] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.550134] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.552326] mousedev: PS/2 mouse device common for all mice [ 3.555643] rtc_cmos 00:05: RTC can wake from S4 [ 3.559659] rtc_cmos 00:05: registered as rtc0 [ 3.561194] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.562519] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.568442] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.568724] intel_pstate: CPU model not supported [ 3.572247] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.576480] hid: raw HID events driver (C) Jiri Kosina [ 3.578758] usbcore: registered new interface driver usbhid [ 3.580968] usbhid: USB HID core driver [ 3.582533] drop_monitor: Initializing network drop monitor service [ 3.584933] Initializing XFRM netlink socket [ 3.590040] NET: Registered protocol family 10 [ 3.592742] Segment Routing with IPv6 [ 3.594189] NET: Registered protocol family 17 [ 3.596074] mpls_gso: MPLS GSO support [ 3.603191] RAS: Correctable Errors collector initialized. [ 3.604766] AVX version of gcm_enc/dec engaged. [ 3.606009] AES CTR mode by8 optimization enabled [ 3.679478] sched_clock: Marking stable (3679451659, 0)->(4545835145, -866383486) [ 3.683477] registered taskstats version 1 [ 3.685106] Loading compiled-in X.509 certificates [ 3.686565] zswap: loaded using pool lzo/zbud [ 3.716420] Key type big_key registered [ 3.729123] Key type encrypted registered [ 3.730523] ima: No TPM chip found, activating TPM-bypass! [ 3.732426] ima: Allocated hash algorithm: sha1 [ 3.733588] ima: No architecture policies found [ 3.734888] evm: Initialising EVM extended attributes: [ 3.736250] evm: security.selinux [ 3.737190] evm: security.ima [ 3.737909] evm: security.capability [ 3.738872] evm: HMAC attrs: 0x1 [ 3.740835] rtc_cmos 00:05: setting system clock to 2026-08-27 14:54:41 UTC (1787842481) [ 3.747436] debug: unmapping init [mem 0xffffffff8ec03000-0xffffffff8edfffff] [ 3.749831] debug: unmapping init [mem 0xffffffff8d982000-0xffffffff8dc58fff] [ 3.759082] Write protecting the kernel read-only data: 28672k [ 3.762050] debug: unmapping init [mem 0xffffffff8c003000-0xffffffff8c1fffff] [ 3.764256] debug: unmapping init [mem 0xffffffff8c914000-0xffffffff8c9fffff] [ 3.797334] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.805925] systemd[1]: Detected virtualization kvm. [ 3.807364] systemd[1]: Detected architecture x86-64. [ 3.808762] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.836386] systemd[1]: No hostname configured. [ 3.838337] systemd[1]: Set hostname to . [ 3.839639] random: systemd: uninitialized urandom read (16 bytes read) [ 3.842050] systemd[1]: Initializing machine ID from random generator. [ 3.890100] random: ln: uninitialized urandom read (6 bytes read) [ 3.982635] random: systemd: uninitialized urandom read (16 bytes read) [ 3.985247] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.989942] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.994219] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Timers. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... Starting Journal Service... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.721872] device-mapper: uevent: version 1.0.3 [ 4.724241] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.562312] virtio_net virtio0 ens2: renamed from eth0 [ 5.577728] random: fast init done [ 5.619154] scsi host0: ata_piix [ 5.634464] scsi host1: ata_piix [ 5.635961] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.638854] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.365840] dracut-initqueue[588]: RTNETLINK answers: File exists [ 10.244469] random: crng init done [ 10.246158] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.777212] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.029732] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.301453] SELinux: Disabled at runtime. [ 12.366045] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.375538] systemd[1]: Detected virtualization kvm. [ 12.377479] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.946601] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.950199] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.957898] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.961801] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.966590] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.982536] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.989498] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. Starting Apply Kernel Variables... [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Control Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target rpc_pipefs.target. Starting udev Coldplug all Devices... Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root a[ 13.083914] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS nd Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Huge Pages File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd File Systems. Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 13.531874] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.951263] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.999318] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.206183] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.216560] EDAC sbridge: Ver: 1.1.2 [ 15.926511] Key type dns_resolver registered [ 16.241813] NFS: Registering the id_resolver key type [ 16.243850] Key type id_resolver registered [ 16.245495] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg626-server login: [ 51.616861] libcfs: loading out-of-tree module taints kernel. [ 51.718202] Key type ._llcrypt registered [ 51.725519] Key type .llcrypt registered [ 51.896858] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_hostid [ 69.400425] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 70.740637] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 70.779447] alg: No test for adler32 (adler32-zlib) [ 72.545582] Lustre: Lustre: Build Version: 2.17.57_92_gffc8991 [ 73.611016] hrtimer: interrupt took 4004419 ns [ 73.644885] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [ 75.511845] Key type lgssc registered [ 77.916539] Lustre: Echo OBD driver; http://www.lustre.org/ [ 98.716649] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 142.705899] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 157.459792] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 157.520686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 158.884447] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 158.913661] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 159.013841] Lustre: lustre-MDT0000: new disk, initializing [ 159.077264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 159.087788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 163.206920] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 177.039165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 177.216564] Lustre: 6518:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 177.267502] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 177.276139] Lustre: Skipped 1 previous similar message [ 177.473357] Lustre: lustre-MDT0001: new disk, initializing [ 177.595351] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 177.639140] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 177.661280] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 182.901084] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 188.417955] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 197.865437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 198.207100] Lustre: lustre-OST0000: new disk, initializing [ 198.214176] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 198.228097] Lustre: 8454:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 198.304848] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 201.958339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 201.966294] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 202.069137] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 204.727962] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 220.179663] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 220.420514] Lustre: lustre-OST0001: new disk, initializing [ 220.427499] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 220.439706] Lustre: 9528:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 220.579074] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 227.500389] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 228.449320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 228.466121] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 228.569946] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 241.091225] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 249.613944] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 256.400787] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing check_logdir /tmp/testlogs/ [ 262.336044] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing yml_node [ 266.866956] Lustre: DEBUG MARKER: Client: 2.17.57.92 [ 269.661149] Lustre: DEBUG MARKER: MDS: 2.17.57.92 [ 272.248408] Lustre: DEBUG MARKER: OSS: 2.17.57.92 [ 273.842859] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 10:59:09 EDT 2026 [ 293.069334] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 294.662684] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 315.880381] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 315.882158] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 315.891703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 315.897738] Lustre: Skipped 1 previous similar message [ 317.837448] Lustre: server umount lustre-MDT0000 complete [ 321.389223] LustreError: 8464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842799 with bad export cookie 7090562689352188234 [ 321.395552] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 321.401965] LustreError: 8464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 326.114320] LustreError: 6522:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 326.114583] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 326.124745] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 326.124757] Lustre: Skipped 3 previous similar messages [ 326.129575] LustreError: 6522:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 326.176931] Lustre: Skipped 3 previous similar messages [ 327.906936] Lustre: server umount lustre-MDT0001 complete [ 331.655892] Lustre: server umount lustre-OST0000 complete [ 335.687116] Lustre: server umount lustre-OST0001 complete [ 341.611135] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_hostid [ 351.859713] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 390.260332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 390.580571] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 390.620665] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 390.711597] Lustre: lustre-MDT0000: new disk, initializing [ 390.805105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 390.852107] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 395.488751] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 407.795294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 408.007661] Lustre: 15092:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 408.077376] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 408.080319] Lustre: Skipped 1 previous similar message [ 408.262199] Lustre: lustre-MDT0001: new disk, initializing [ 408.381185] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 408.468963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 408.476756] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 412.645420] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 422.324910] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 427.965739] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 434.482246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 434.836685] Lustre: lustre-OST0000: new disk, initializing [ 434.843457] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 434.849542] Lustre: 17064:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 434.954969] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 436.048566] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 436.057143] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 436.145506] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 440.396242] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 449.420733] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 453.437856] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 453.647990] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 457.398919] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 457.565983] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 461.796231] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 461.802799] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 461.818799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 461.819693] LustreError: Skipped 1 previous similar message [ 461.835321] Lustre: Skipped 2 previous similar messages [ 465.523253] Lustre: server umount lustre-OST0000 complete [ 469.985044] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 470.001302] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 470.023506] Lustre: Skipped 1 previous similar message [ 470.032184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 477.152585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 477.158362] Lustre: Skipped 3 previous similar messages [ 483.295354] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 483.620437] Lustre: server umount lustre-MDT0000 complete [ 486.368374] LustreError: 15104:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 486.403780] LustreError: 15104:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 488.393901] LustreError: 15086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842966 with bad export cookie 7090562689352189767 [ 488.406806] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 488.428258] LustreError: 15086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 488.972558] Lustre: server umount lustre-MDT0001 complete [ 495.646075] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 11:02:51 (1787842971) [ 514.375935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 515.060525] LustreError: 19938:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 515.146426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 519.418840] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 520.162480] LustreError: 19939:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 524.274773] LustreError: 19938:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 529.374845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 534.291197] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 542.265127] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 548.526630] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 550.598679] Lustre: Failing over lustre-MDT0000 [ 550.903663] Lustre: server umount lustre-MDT0000 complete [ 554.727520] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 554.730550] LustreError: 19934:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 554.746751] Lustre: Skipped 2 previous similar messages [ 554.773657] LustreError: 19934:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 570.335264] Lustre: 3649:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787843032/real 1787843032] req@ffff90d243ae43c0 x1874688790918528/t0(0) o400->MGC192.168.206.126@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787843048 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 570.364442] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 570.569342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 574.504204] LustreError: 19934:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 574.524424] LustreError: 19934:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [ 581.601114] LustreError: 3647:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff90d360e48780 x1874688790921472/t0(0) o250->MGC192.168.206.126@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 581.890069] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 581.897527] Lustre: Skipped 1 previous similar message [ 586.424844] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 587.233412] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 587.247244] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 587.273614] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 595.707767] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 600.354275] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 600.358520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 600.373374] Lustre: Skipped 1 previous similar message [ 600.379285] Lustre: Skipped 2 previous similar messages [ 605.827919] Lustre: server umount lustre-MDT0000 complete [ 610.787835] LustreError: 20951:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 610.818645] LustreError: 20951:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 9 previous similar messages [ 610.878888] LustreError: 19920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843088 with bad export cookie 7090562689352191181 [ 610.886933] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 610.889425] LustreError: 19920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 611.444714] Lustre: server umount lustre-MDT0001 complete [ 622.175042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 627.612569] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 636.316695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 641.639227] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 650.213711] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 656.279759] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 658.131166] Lustre: Failing over lustre-MDT0000 [ 658.402978] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 658.465746] Lustre: server umount lustre-MDT0000 complete [ 676.512123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 676.696934] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 676.834259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 676.840945] Lustre: Skipped 1 previous similar message [ 677.047862] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 681.471710] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 682.093646] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 682.110899] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 682.113967] Lustre: Skipped 1 previous similar message [ 682.134897] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 689.638724] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 693.344563] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 693.369028] Lustre: Skipped 2 previous similar messages [ 699.790527] Lustre: server umount lustre-MDT0000 complete [ 703.192786] LustreError: 23263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843180 with bad export cookie 7090562689352192119 [ 703.194163] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 703.201597] LustreError: 23263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 703.578502] Lustre: server umount lustre-MDT0001 complete [ 711.397941] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 11:06:27 (1787843187) [ 726.119287] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_hostid [ 735.037431] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 774.020401] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 783.844829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 784.067155] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 784.090335] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 784.150389] Lustre: lustre-MDT0000: new disk, initializing [ 784.206263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 784.210747] Lustre: Skipped 3 previous similar messages [ 784.229789] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 787.720654] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 797.386355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 797.540689] Lustre: 29653:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 797.907096] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 802.244144] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 810.939240] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 816.804972] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 823.474939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 823.720688] Lustre: 31631:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 825.042132] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 825.078095] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 825.121650] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 825.144508] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 830.308119] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 839.285651] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 844.500986] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 866.281791] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 866.282106] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 866.288810] LustreError: Skipped 1 previous similar message [ 866.297488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 866.306103] Lustre: Skipped 1 previous similar message [ 866.341989] Lustre: Skipped 5 previous similar messages [ 870.746351] Lustre: server umount lustre-OST0000 complete [ 874.978817] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 880.941092] Lustre: server umount lustre-MDT0000 complete [ 881.633749] LustreError: 30526:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 881.668724] LustreError: 30526:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 12 previous similar messages [ 885.101282] LustreError: 29646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843362 with bad export cookie 7090562689352192525 [ 885.104183] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 885.114109] LustreError: 29646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 885.741597] Lustre: server umount lustre-MDT0001 complete [ 897.778357] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [ 900.937954] Key type lgssc unregistered [ 901.281761] LNet: 33612:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 901.295414] LNetError: 33612:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 901.321184] LNet: Removed LNI 192.168.206.126@tcp [ 902.275189] Key type .llcrypt unregistered [ 902.276689] Key type ._llcrypt unregistered [ 925.721609] Key type ._llcrypt registered [ 925.725681] Key type .llcrypt registered [ 925.848610] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 926.703149] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 926.741172] alg: No test for adler32 (adler32-zlib) [ 927.769718] Lustre: Lustre: Build Version: 2.17.57_92_gffc8991 [ 927.988332] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [ 929.655377] Key type lgssc registered [ 930.679265] Lustre: Echo OBD driver; http://www.lustre.org/ [ 937.308677] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 937.329343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 938.744757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 943.423845] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 949.218153] LustreError: 35027:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 949.239516] LustreError: 35027:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 951.897385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 952.152696] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 956.051669] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 962.492630] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 968.163799] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 974.347932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 974.628861] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 976.687651] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 979.627760] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 983.867572] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 985.675060] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 997.180351] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 11:11:12 (1787843472) [ 999.080670] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1000.686279] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1006.396027] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 11:11:22 (1787843482) [ 1015.721113] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 11:11:31 (1787843491) [ 1016.702763] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1025.631732] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1025.638292] Lustre: Skipped 2 previous similar messages [ 1027.153608] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 11:11:42 (1787843502) [ 1030.892934] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1030.899459] Lustre: Skipped 3 previous similar messages [ 1047.695902] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1047.700455] Lustre: Skipped 6 previous similar messages [ 1050.634762] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 11:12:05 (1787843525) [ 1052.963879] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1056.258701] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1063.818673] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 11:12:19 (1787843539) [ 1065.553213] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1065.557377] Lustre: Skipped 3 previous similar messages [ 1076.490627] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 11:12:31 (1787843551) [ 1091.553826] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1091.557815] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1091.561026] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1091.574122] Lustre: Skipped 1 previous similar message [ 1096.684023] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1096.693709] Lustre: Skipped 2 previous similar messages [ 1101.794459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1104.355606] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1104.500546] Lustre: server umount lustre-OST0000 complete [ 1111.012113] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1111.023460] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1111.035139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1111.040158] Lustre: Skipped 1 previous similar message [ 1112.034072] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1114.396202] Lustre: server umount lustre-MDT0000 complete [ 1117.163493] LustreError: 35756:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1117.178718] LustreError: 35756:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1117.514342] LustreError: 35007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843595 with bad export cookie 13687555059988693761 [ 1117.525461] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1117.797441] Lustre: server umount lustre-MDT0001 complete [ 1126.339992] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [ 1128.455967] Key type lgssc unregistered [ 1128.665962] LNet: 42869:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1128.677543] LNetError: 42869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1128.690863] LNet: Removed LNI 192.168.206.126@tcp [ 1129.267206] Key type .llcrypt unregistered [ 1129.269780] Key type ._llcrypt unregistered [ 1142.567489] Key type ._llcrypt registered [ 1142.571924] Key type .llcrypt registered [ 1142.692594] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_hostid [ 1156.048284] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 1157.120199] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1157.146085] alg: No test for adler32 (adler32-zlib) [ 1158.194457] Lustre: Lustre: Build Version: 2.17.57_92_gffc8991 [ 1158.537045] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [ 1160.271176] Key type lgssc registered [ 1161.207476] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1194.491541] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1194.522327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1195.840129] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1195.884579] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1196.007782] Lustre: lustre-MDT0000: new disk, initializing [ 1196.089141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1196.111907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1200.903140] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 1211.594931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1211.706693] Lustre: 46017:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1211.746299] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1211.749892] Lustre: Skipped 1 previous similar message [ 1211.827365] Lustre: lustre-MDT0001: new disk, initializing [ 1211.938095] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1211.987629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1211.992592] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1216.462296] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 1226.812972] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1232.874473] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1239.807811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1240.184168] Lustre: lustre-OST0000: new disk, initializing [ 1240.189648] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1240.206405] Lustre: 47987:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1240.318635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1246.597658] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 1247.786710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1247.803954] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1247.938733] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1256.005443] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1260.948385] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1261.205965] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1265.364783] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1265.529086] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1268.714958] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1268.716483] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1268.719871] LustreError: Skipped 1 previous similar message [ 1268.724419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1268.759233] Lustre: Skipped 1 previous similar message [ 1273.793208] Lustre: server umount lustre-OST0000 complete [ 1278.434955] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1278.448196] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1278.466279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1278.472338] Lustre: Skipped 1 previous similar message [ 1283.687183] Lustre: server umount lustre-MDT0000 complete [ 1287.819304] LustreError: 46011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843765 with bad export cookie 2327120254643665113 [ 1287.823471] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1287.834308] LustreError: 46011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1288.500599] Lustre: server umount lustre-MDT0001 complete [ 1299.967550] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 11:16:14 (1787843774) [ 1307.426036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1307.957266] LustreError: 50448:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1307.972869] LustreError: 50448:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 1308.027308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1313.251034] LustreError: 50449:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1313.273824] LustreError: 50449:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1313.359824] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 1317.359748] LustreError: 50448:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1321.914676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1322.196214] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1326.426360] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 1333.632666] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1339.400214] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1346.640600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1347.067150] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1353.931720] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 1361.292688] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1365.810963] LustreError: 52977:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1366.620466] Lustre: Modifying parameter general.timeout=1 in log params [ 1368.190398] Lustre: Modifying parameter general.timeout=3 in log params [ 1368.195834] Lustre: Skipped 1 previous similar message [ 1370.985808] Lustre: Modifying parameter general.timeout=6 in log params [ 1370.992281] Lustre: Skipped 2 previous similar messages [ 1375.149402] Lustre: Modifying parameter general.timeout=11 in log params [ 1375.154274] Lustre: Skipped 4 previous similar messages [ 1383.210198] Lustre: Modifying parameter general.timeout=20 in log params [ 1383.218256] Lustre: Skipped 8 previous similar messages [ 1399.223450] Lustre: Modifying parameter general.timeout=40 in log params [ 1399.228993] Lustre: Skipped 19 previous similar messages [ 1431.675855] Lustre: Modifying parameter general.timeout=77 in log params [ 1431.682157] Lustre: Skipped 36 previous similar messages [ 1490.113362] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 11:19:25 (1787843965) [ 1492.991183] Lustre: 59918:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1494.713458] Lustre: 60030:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1495.798959] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 in log params [ 1495.808182] Lustre: Skipped 56 previous similar messages [ 1496.960626] Lustre: 60129:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1503.217494] Lustre: 60424:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1503.241257] Lustre: 60424:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1531.880991] Lustre: 61800:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1531.912826] LustreError: 61800:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1535.585091] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 11:20:11 (1787844011) [ 1547.744617] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1547.756565] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1547.769380] Lustre: Skipped 1 previous similar message [ 1547.779521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1547.788078] Lustre: Skipped 1 previous similar message [ 1550.822135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1550.822774] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1550.831812] Lustre: Skipped 1 previous similar message [ 1550.855326] Lustre: Skipped 1 previous similar message [ 1552.823590] Lustre: server umount lustre-MDT0000 complete [ 1555.924180] LustreError: 51184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844033 with bad export cookie 2327120254643665995 [ 1555.937195] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1555.950968] LustreError: 50443:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1555.953276] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1555.972067] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1556.304705] Lustre: server umount lustre-MDT0001 complete [ 1566.211380] Lustre: server umount lustre-OST0000 complete [ 1592.916848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1593.229328] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1593.473426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1598.103170] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 1606.406350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1606.592530] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1606.617567] Lustre: 64330:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1612.248976] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 1620.191186] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1626.512911] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1633.609414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1633.872308] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1634.331191] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1634.343069] Lustre: Skipped 1 previous similar message [ 1641.308632] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 1649.118312] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1650.776974] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1650.788952] Lustre: Skipped 19 previous similar messages [ 1650.827572] Lustre: 66896:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1660.907987] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1660.916383] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1660.933961] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1663.567728] Lustre: server umount lustre-OST0000 complete [ 1668.582839] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1668.593060] LustreError: Skipped 1 previous similar message [ 1673.724473] Lustre: server umount lustre-MDT0000 complete [ 1676.771297] LustreError: 64337:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1676.800169] LustreError: 64337:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1677.839073] LustreError: 64324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844155 with bad export cookie 2327120254643669047 [ 1677.846600] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1677.860098] LustreError: 64324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 1678.240322] Lustre: server umount lustre-MDT0001 complete [ 1690.187088] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [ 1693.162112] Key type lgssc unregistered [ 1693.535278] LNet: 68106:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1693.548163] LNetError: 68106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1693.578467] LNet: Removed LNI 192.168.206.126@tcp [ 1694.406094] Key type .llcrypt unregistered [ 1694.409040] Key type ._llcrypt unregistered [ 1717.566765] Key type ._llcrypt registered [ 1717.574200] Key type .llcrypt registered [ 1717.690587] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 1718.663530] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1718.729701] alg: No test for adler32 (adler32-zlib) [ 1719.763914] Lustre: Lustre: Build Version: 2.17.57_92_gffc8991 [ 1720.021074] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [ 1721.663703] Key type lgssc registered [ 1722.741932] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1730.029550] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1730.043552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1731.587571] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1735.995956] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 1741.803058] LustreError: 69519:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1741.820783] LustreError: 69519:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1743.753899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1744.355885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1748.365959] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 1755.736636] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1760.940818] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1767.723500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1768.348459] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1774.144675] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 1781.450657] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1788.693493] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1797.092783] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1797.095830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1797.115701] Lustre: Skipped 1 previous similar message [ 1802.208823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1802.225403] Lustre: Skipped 3 previous similar messages [ 1803.118769] Lustre: server umount lustre-MDT0000 complete [ 1806.797182] LustreError: 69500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844284 with bad export cookie 9075288332427030411 [ 1806.805324] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1806.808109] LustreError: 69500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1807.210275] Lustre: server umount lustre-MDT0001 complete [ 1820.863643] Lustre: server umount lustre-OST0000 complete [ 1844.957256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1845.224809] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1845.264747] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1845.539509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1850.193995] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 1857.446564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1857.516855] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1857.541078] Lustre: 74389:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1858.011887] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1861.804682] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 1868.031734] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1873.917686] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1880.712157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1881.094434] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1881.716763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1888.100253] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 1894.982541] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1896.341084] Lustre: 76964:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1897.943668] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1901.536342] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1901.544500] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1901.548437] LustreError: Skipped 1 previous similar message [ 1901.550608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1901.587923] Lustre: Skipped 2 previous similar messages [ 1906.662229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1906.666467] Lustre: Skipped 2 previous similar messages [ 1907.367572] Lustre: server umount lustre-OST0000 complete [ 1911.776271] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1911.781638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1911.786740] Lustre: Skipped 1 previous similar message [ 1911.815620] Lustre: Skipped 1 previous similar message [ 1916.772370] Lustre: server umount lustre-MDT0000 complete [ 1916.902245] LustreError: 74396:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1916.928781] LustreError: 74396:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1920.246797] LustreError: 74380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844398 with bad export cookie 9075288332427031244 [ 1920.251912] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1920.260074] LustreError: 74380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1920.510429] Lustre: server umount lustre-MDT0001 complete [ 1929.489915] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [ 1931.985500] Key type lgssc unregistered [ 1932.297741] LNet: 78178:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1932.308763] LNetError: 78178:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1932.340177] LNet: Removed LNI 192.168.206.126@tcp [ 1933.101338] Key type .llcrypt unregistered [ 1933.104446] Key type ._llcrypt unregistered [ 1953.498696] Key type ._llcrypt registered [ 1953.500463] Key type .llcrypt registered [ 1953.609821] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 1954.460549] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1954.478874] alg: No test for adler32 (adler32-zlib) [ 1955.636401] Lustre: Lustre: Build Version: 2.17.57_92_gffc8991 [ 1955.897868] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [ 1957.623213] Key type lgssc registered [ 1958.747890] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1965.293826] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1965.308850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1966.689154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1970.446367] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 1976.804826] LustreError: 79570:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1976.822880] LustreError: 79570:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1978.289661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1978.783084] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1982.245221] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 1988.686624] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1993.179728] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1999.396202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2000.027976] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2006.104222] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 2012.312575] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2017.903160] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2018.512589] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2020.071630] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2020.083794] Lustre: Skipped 1 previous similar message [ 2022.299209] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2022.303098] Lustre: Skipped 2 previous similar messages [ 2026.872648] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2026.880238] Lustre: Skipped 5 previous similar messages [ 2035.379391] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2035.383310] Lustre: Skipped 9 previous similar messages [ 2051.516477] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2051.521035] Lustre: Skipped 16 previous similar messages [ 2083.955269] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2083.963651] Lustre: Skipped 39 previous similar messages [ 2109.907392] Lustre: 87667:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2112.710962] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 11:29:48 (1787844588) [ 2114.253585] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2116.010243] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 11:29:51 (1787844591) [ 2122.217973] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2122.222584] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2122.230698] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2124.256756] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2124.256981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2124.274884] Lustre: Skipped 1 previous similar message [ 2126.140534] Lustre: server umount lustre-OST0000 complete [ 2132.454308] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2132.461822] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2132.474544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2134.497721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2134.501804] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2136.489954] Lustre: server umount lustre-MDT0000 complete [ 2139.620641] LustreError: 79567:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2139.642691] LustreError: 79567:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2140.604168] LustreError: 79552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844618 with bad export cookie 8579106611727323476 [ 2140.613124] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2140.617480] LustreError: 79552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2141.013931] Lustre: server umount lustre-MDT0001 complete [ 2151.323540] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [ 2153.940904] Key type lgssc unregistered [ 2154.282884] LNet: 88919:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2154.301565] LNetError: 88919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2154.319483] LNet: Removed LNI 192.168.206.126@tcp [ 2155.165336] Key type .llcrypt unregistered [ 2155.168876] Key type ._llcrypt unregistered [ 2159.738680] Key type ._llcrypt registered [ 2159.740858] Key type .llcrypt registered [ 2159.849873] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2163.963215] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules [ 2164.807171] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2164.858146] alg: No test for adler32 (adler32-zlib) [ 2165.949827] Lustre: Lustre: Build Version: 2.17.57_92_gffc8991 [ 2166.194271] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [ 2167.903990] Key type lgssc registered [ 2168.987307] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2170.032567] LustreError: 90122:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2186.497829] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 2191.945080] LustreError: 90122:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2194.168248] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2194.183659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2195.613891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2199.443718] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 2206.176447] LustreError: 91144:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2206.189905] LustreError: 91144:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2207.319783] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 11:31:23 (1787844683) [ 2211.296394] LustreError: 91143:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2211.908302] LustreError: 91175:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2211.917452] LustreError: 91175:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2211.926810] LustreError: 91175:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 0, failed: rc = -5 [ 2212.276177] Lustre: server umount lustre-MDT0000 complete [ 2224.005632] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [ 2226.157316] Key type lgssc unregistered [ 2226.397887] LNet: 92672:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2226.404088] LNetError: 92672:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2226.417605] LNet: Removed LNI 192.168.206.126@tcp [ 2227.023440] Key type .llcrypt unregistered [ 2227.026614] Key type ._llcrypt unregistered [ 2240.521210] Key type ._llcrypt registered [ 2240.522773] Key type .llcrypt registered [ 2240.596101] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 2241.395984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2241.415467] alg: No test for adler32 (adler32-zlib) [ 2242.377290] Lustre: Lustre: Build Version: 2.17.57_92_gffc8991 [ 2242.584822] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [ 2244.247375] Key type lgssc registered [ 2245.323090] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2252.206639] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2252.226806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2253.656833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2257.706562] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 2264.033686] LustreError: 93870:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2264.049382] LustreError: 93870:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2265.972167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2266.675293] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2270.802189] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 2277.568727] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2282.643970] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2288.201927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2288.762654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2294.652310] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 2301.198726] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2345.813245] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 11:33:41 (1787844821) [ 2347.792093] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2349.780993] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 11:33:45 (1787844825) [ 2355.686722] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2355.697608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2355.707843] Lustre: Skipped 1 previous similar message [ 2360.364257] Lustre: server umount lustre-MDT0000 complete [ 2360.807518] LustreError: 93866:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2360.830958] LustreError: 93866:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2363.778218] LustreError: 93850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844841 with bad export cookie 15645186747025057802 [ 2363.778729] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2363.783613] LustreError: 93850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2364.067550] Lustre: server umount lustre-MDT0001 complete [ 2379.412606] Lustre: server umount lustre-OST0000 complete [ 2395.886506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2396.464368] LustreError: 98147:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2396.665764] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2401.212844] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 2401.777099] LustreError: 98148:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2406.881685] LustreError: 98147:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2410.183323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2410.845847] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2415.286101] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 2422.764500] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2427.770887] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2439.764038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2440.034763] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2440.040744] LustreError: 98137:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2440.090065] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2440.101742] LustreError: 100229:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2440.108731] LustreError: 100229:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2440.112188] LustreError: 100229:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2440.175805] Lustre: server umount lustre-OST0000 complete [ 2440.178098] LustreError: 100229:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2449.062304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2449.229259] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2449.246949] LustreError: 98136:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2449.275605] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2449.286134] LustreError: 100697:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2449.293897] LustreError: 100697:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2449.298881] LustreError: 100697:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2449.378425] Lustre: server umount lustre-OST0000 complete [ 2449.386599] LustreError: 100697:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2458.203661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2458.323843] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2458.328143] Lustre: Found index 0 for lustre-OST0000, updating log [ 2458.332873] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2458.406306] Lustre: lustre-OST0000: new disk, initializing [ 2458.411461] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2458.422726] Lustre: 101233:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2458.715336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2463.832040] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 2467.471379] Lustre: Failing over lustre-OST0000 [ 2467.628746] Lustre: server umount lustre-OST0000 complete [ 2471.905696] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2471.912818] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2471.923657] Lustre: Skipped 1 previous similar message [ 2471.929119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2471.934015] Lustre: Skipped 2 previous similar messages [ 2473.955533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2473.956987] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2479.076831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2479.083511] Lustre: Skipped 2 previous similar messages [ 2485.216982] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2485.465227] Lustre: server umount lustre-MDT0000 complete [ 2486.248410] LustreError: 98147:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2486.267631] LustreError: 98147:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2488.915922] LustreError: 101231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844966 with bad export cookie 15645186747025060546 [ 2488.922208] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2488.923796] LustreError: 101231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2489.163778] Lustre: server umount lustre-MDT0001 complete [ 2497.278351] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 11:36:13 (1787844973) [ 2512.137534] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 2522.155791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2522.568989] LustreError: 103577:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2522.711527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2526.839245] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 2534.489486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2538.671260] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 2541.581662] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2547.396727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2553.850404] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 2562.596434] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2562.696777] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2562.805391] Lustre: lustre-OST0001: new disk, initializing [ 2562.810496] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2562.818958] Lustre: 105895:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2563.166785] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2563.171985] Lustre: Skipped 2 previous similar messages [ 2570.401655] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 2570.469435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2570.487557] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2570.584949] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2578.289878] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2583.005625] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2589.108386] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.26@tcp (stopping) [ 2589.115540] Lustre: Skipped 2 previous similar messages [ 2591.206958] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2591.217120] Lustre: Skipped 2 previous similar messages [ 2594.161391] Lustre: server umount lustre-MDT0001 complete [ 2594.232778] LustreError: 103573:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2594.251112] LustreError: 103573:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2604.587902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2604.739108] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2604.746192] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2604.799772] Lustre: 103567:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2606.591198] LustreError: 107466:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2606.598385] LustreError: 107466:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2606.617250] LustreError: 103569:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2606.732076] LustreError: 107480:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.126@tcp: cfg command failed: rc = -17 [ 2606.738457] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2606.738457] [ 2610.410412] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 2610.680346] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2620.348114] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 11:38:15 (1787845095) [ 2638.522988] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 2657.537606] Lustre: 109512:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2680.151511] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2683.812426] Lustre: 110646:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2763.743929] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2763.754611] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2763.776649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2763.781778] Lustre: Skipped 3 previous similar messages [ 2769.279414] Lustre: server umount lustre-MDT0000 complete [ 2769.382872] LustreError: 103573:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2769.411114] LustreError: 103573:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 11 previous similar messages [ 2773.443064] LustreError: 106614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845251 with bad export cookie 15645186747025061540 [ 2773.443303] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2773.474824] LustreError: 106614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2773.742531] Lustre: server umount lustre-MDT0001 complete [ 2787.402708] Lustre: server umount lustre-OST0000 complete [ 2800.784865] Lustre: server umount lustre-OST0001 complete [ 2806.687241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2817.130436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2826.073069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2837.386704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2857.246596] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 2866.332482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2866.355128] Lustre: lustre-MDT0000: reset Object Index mappings [ 2866.894443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2866.899457] Lustre: Skipped 1 previous similar message [ 2872.155827] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 2881.768744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2881.797743] Lustre: lustre-MDT0001: reset Object Index mappings [ 2886.864686] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 2889.381690] Lustre: 115745:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2895.790442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2906.140459] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 2908.070210] LustreError: 116100:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2908.086909] LustreError: 116100:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 2908.116423] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 2912.228577] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:354 to 0x280000401:386) [ 2918.543593] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2924.545185] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:195 to 0x2c0000400:225) [ 2924.558231] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 2925.741887] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 2933.727396] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2936.999137] Lustre: 117626:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2980.495881] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 11:44:16 (1787845456) [ 3010.535527] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3010.548959] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3010.576954] Lustre: Skipped 3 previous similar messages [ 3010.592461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3010.599745] Lustre: Skipped 4 previous similar messages [ 3013.742859] Lustre: server umount lustre-MDT0000 complete [ 3017.421843] LustreError: 116130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845495 with bad export cookie 15645186747025108391 [ 3017.439185] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3023.904910] Lustre: server umount lustre-MDT0001 complete [ 3033.468817] Lustre: server umount lustre-OST0000 complete [ 3042.697397] Lustre: server umount lustre-OST0001 complete [ 3050.483361] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_hostid [ 3060.935096] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 3092.321719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3092.525850] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3092.547240] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3092.607968] Lustre: lustre-MDT0000: new disk, initializing [ 3092.660298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3092.664633] Lustre: Skipped 3 previous similar messages [ 3092.676198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3096.551530] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 3102.575016] Lustre: server umount lustre-MDT0000 complete [ 3111.113152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3111.361278] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3111.375544] Lustre: 121889:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3111.401971] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3115.662918] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 3119.921826] Lustre: server umount lustre-MDT0000 complete [ 3127.078394] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 11:46:42 (1787845602) [ 3128.527890] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3130.349688] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 11:46:45 (1787845605) [ 3146.588695] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 3155.465283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3159.499981] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 3166.878596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3166.970586] Lustre: 123940:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3167.033325] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3167.036236] Lustre: Skipped 1 previous similar message [ 3167.097842] Lustre: lustre-MDT0001: new disk, initializing [ 3167.268764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3167.284860] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3171.332809] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 3175.593659] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3181.286044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3181.553732] Lustre: lustre-OST0000: new disk, initializing [ 3181.560043] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3181.569528] Lustre: 125516:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3182.852493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3182.863643] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3182.988488] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3187.083914] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 3196.891931] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3197.062674] Lustre: lustre-OST0001: new disk, initializing [ 3197.066990] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3197.074736] Lustre: 126385:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3199.165072] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3199.171711] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3199.289594] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3203.306517] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 3211.833958] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3215.049595] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3581.932456] Lustre: 132798:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3581.937426] Lustre: 132798:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3581.941527] Lustre: 132798:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3581.945800] Lustre: 132798:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3581.950260] Lustre: 132798:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3581.954335] Lustre: 132798:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3857.722752] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 11:58:52 (1787846332) [ 3873.762934] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3873.779417] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3873.798943] Lustre: Skipped 5 previous similar messages [ 3873.804671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3873.812944] Lustre: Skipped 8 previous similar messages [ 3878.910416] Lustre: server umount lustre-MDT0000 complete [ 3880.418606] LustreError: 123945:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3880.439355] LustreError: 123945:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 15 previous similar messages [ 3882.284029] LustreError: 132837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846360 with bad export cookie 15645186747025175115 [ 3882.287814] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3882.298578] LustreError: 132837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 3885.536875] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3885.552603] Lustre: Skipped 4 previous similar messages [ 3885.553900] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3885.561676] Lustre: Skipped 5 previous similar messages [ 3888.922538] Lustre: server umount lustre-MDT0001 complete [ 3898.533591] Lustre: server umount lustre-OST0000 complete [ 3908.346994] Lustre: server umount lustre-OST0001 complete [ 3915.463292] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_hostid [ 3925.289130] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 3960.100235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3960.291760] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3960.343899] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3960.444413] Lustre: lustre-MDT0000: new disk, initializing [ 3960.538202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3960.549490] Lustre: Skipped 5 previous similar messages [ 3960.571515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3965.267636] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 3975.168339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3975.296063] Lustre: 135843:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3975.346590] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3975.357085] Lustre: Skipped 1 previous similar message [ 3975.412542] Lustre: lustre-MDT0001: new disk, initializing [ 3975.533464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3975.542551] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3980.888400] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 3991.461136] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3997.263062] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4004.381987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4004.701963] Lustre: lustre-OST0000: new disk, initializing [ 4004.705362] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4004.710538] Lustre: 137828:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4004.790770] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4004.795366] Lustre: Skipped 1 previous similar message [ 4006.095555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4006.105051] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4006.245721] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4011.200513] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 4021.401064] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4031.225707] Lustre: lustre-MDD0000: changelog on [ 4034.022813] Lustre: lustre-MDD0001: changelog on [ 5709.350823] Lustre: lustre-MDD0001: changelog off [ 5711.437797] Lustre: 139283:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5712.103825] Lustre: lustre-MDD0000: changelog off [ 5721.578784] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5721.580320] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5721.584787] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5721.584796] Lustre: Skipped 2 previous similar messages [ 5721.601472] Lustre: Skipped 1 previous similar message [ 5721.633919] LustreError: Skipped 1 previous similar message [ 5723.779615] Lustre: server umount lustre-OST0000 complete [ 5731.810330] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5731.811493] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5731.836533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5731.849431] Lustre: Skipped 1 previous similar message [ 5731.880718] Lustre: Skipped 2 previous similar messages [ 5734.527789] Lustre: server umount lustre-MDT0000 complete [ 5736.929519] LustreError: 135857:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5736.950944] LustreError: 135857:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 5738.666454] LustreError: 137827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848216 with bad export cookie 15645186747025991287 [ 5738.669182] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5738.676457] LustreError: 137827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5739.044184] Lustre: server umount lustre-MDT0001 complete [ 5750.779908] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [ 5754.038634] Key type lgssc unregistered [ 5754.451221] LNet: 146933:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5754.462579] LNetError: 146933:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5754.485435] LNet: Removed LNI 192.168.206.126@tcp [ 5755.550187] Key type .llcrypt unregistered [ 5755.554771] Key type ._llcrypt unregistered [ 5776.172655] Key type ._llcrypt registered [ 5776.177180] Key type .llcrypt registered [ 5776.303348] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_hostid [ 5791.483944] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 5792.673513] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5792.724506] alg: No test for adler32 (adler32-zlib) [ 5793.896307] Lustre: Lustre: Build Version: 2.17.57_92_gffc8991 [ 5794.178833] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [ 5795.856149] Key type lgssc registered [ 5796.979593] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5831.814418] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5831.841131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5833.128778] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5833.191088] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5833.395371] Lustre: lustre-MDT0000: new disk, initializing [ 5833.539522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5833.601401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5838.669995] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 5849.034417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5849.231169] Lustre: 150136:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5849.296271] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5849.302363] Lustre: Skipped 1 previous similar message [ 5849.430322] Lustre: lustre-MDT0001: new disk, initializing [ 5849.575261] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5849.620782] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5849.628706] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5853.831696] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 5863.380293] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5869.279284] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5876.637371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5876.948239] Lustre: lustre-OST0000: new disk, initializing [ 5876.952403] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5876.960420] Lustre: 152106:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5877.048160] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5882.939532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5882.952505] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5883.074891] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5883.923780] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 5893.378436] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5899.507429] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_module obdecho/obdecho [ 5901.237163] LustreError: 153108:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 5901.251499] LustreError: 153108:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 5902.947287] LustreError: 153204:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 5916.136514] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5916.144892] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5916.158743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5919.206628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5919.219471] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5919.237841] Lustre: Skipped 1 previous similar message [ 5920.635416] Lustre: server umount lustre-MDT0000 complete [ 5924.025858] LustreError: 150128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848401 with bad export cookie 4120435349389602206 [ 5924.032579] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5924.037412] LustreError: 150128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5924.320102] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5924.539680] Lustre: server umount lustre-MDT0001 complete [ 5930.583632] Lustre: server umount lustre-OST0000 complete [ 5952.904015] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_hostid [ 5963.108377] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 5999.067853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5999.310165] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5999.362823] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5999.492776] Lustre: lustre-MDT0000: new disk, initializing [ 5999.588188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5999.601168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6003.209482] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 6013.691664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6013.967910] Lustre: 156916:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6014.083315] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6014.091684] Lustre: Skipped 1 previous similar message [ 6014.327128] Lustre: lustre-MDT0001: new disk, initializing [ 6014.532592] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6014.612199] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6014.623444] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6019.413293] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 6029.093283] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6034.404182] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6040.761587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6041.021947] Lustre: lustre-OST0000: new disk, initializing [ 6041.025619] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6041.031900] Lustre: 158884:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6041.120657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6047.098806] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 6049.336214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6049.350388] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6049.460420] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6055.959643] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6060.657136] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6060.884997] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6065.499337] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6065.752326] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6069.222168] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6069.227294] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6069.235501] Lustre: Skipped 1 previous similar message [ 6069.254029] Lustre: Skipped 3 previous similar messages [ 6073.974937] Lustre: server umount lustre-OST0000 complete [ 6079.457348] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6079.474212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6079.485476] Lustre: Skipped 1 previous similar message [ 6084.577753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6084.583684] Lustre: Skipped 1 previous similar message [ 6092.256776] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6092.440397] Lustre: server umount lustre-MDT0000 complete [ 6094.821202] LustreError: 156923:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6094.843533] LustreError: 156923:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 6096.078980] LustreError: 156909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848573 with bad export cookie 4120435349389603165 [ 6096.082331] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6096.089834] LustreError: 156909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6096.384054] Lustre: server umount lustre-MDT0001 complete [ 6099.990150] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 12:36:15 (1787848575) [ 6101.815774] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6103.391197] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 12:36:19 (1787848579) [ 6110.085097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6110.566512] LustreError: 161296:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6110.668479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6115.506957] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 6115.820403] LustreError: 161297:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6120.930095] LustreError: 161296:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6123.841772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6129.277532] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 6136.511923] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6142.795815] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6149.917559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6150.310635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6150.322299] Lustre: Skipped 1 previous similar message [ 6156.025590] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 6163.373776] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6174.124930] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.26@tcp (stopping) [ 6174.138329] Lustre: Skipped 2 previous similar messages [ 6174.186292] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6174.203612] Lustre: Skipped 3 previous similar messages [ 6175.610643] Lustre: server umount lustre-MDT0000 complete [ 6179.221936] LustreError: 163436:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6179.284448] LustreError: 161277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848657 with bad export cookie 4120435349389604047 [ 6179.289165] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6179.308462] LustreError: 161277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6185.795777] Lustre: server umount lustre-MDT0001 complete [ 6191.756312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6204.220165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6204.561505] LustreError: 165139:0:(llog.c:1655:llog_backup()) MGC192.168.206.126@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6204.572228] Lustre: 165139:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.126@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6204.821179] LustreError: 165144:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6204.854754] LustreError: 165144:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 6224.354082] LustreError: 165146:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6224.372038] LustreError: 165146:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 14 previous similar messages [ 6260.705113] LustreError: 165145:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6260.729417] LustreError: 165145:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 26 previous similar messages [ 6265.312478] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6265.316664] Lustre: 165139:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.126@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6306.293942] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6310.930610] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 6311.402866] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6319.313852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6324.705852] LustreError: 148358:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000cc7936a7 x1874694788741376/t0(0) o250->MGC192.168.206.126@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6324.739620] LustreError: 165146:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6324.771415] LustreError: 165146:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 45 previous similar messages [ 6330.052986] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 6330.352786] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6337.815474] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6339.408920] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6344.755930] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6346.210302] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6366.181058] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6366.185755] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6366.201954] Lustre: Skipped 1 previous similar message [ 6366.207354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6366.228753] Lustre: Skipped 8 previous similar messages [ 6369.482750] Lustre: server umount lustre-MDT0000 complete [ 6373.677376] LustreError: 164081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848851 with bad export cookie 4120435349389610501 [ 6373.680462] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6373.699269] LustreError: 164081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6374.151916] Lustre: server umount lustre-MDT0001 complete [ 6388.939651] Lustre: server umount lustre-OST0000 complete [ 6410.664990] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_hostid [ 6422.705382] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 6460.362783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6460.635415] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6460.678732] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6460.778813] Lustre: lustre-MDT0000: new disk, initializing [ 6460.917204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6460.925792] Lustre: Skipped 1 previous similar message [ 6460.948179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6466.526825] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 6477.113908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6477.247659] Lustre: 170244:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6477.511426] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6482.601921] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 6492.199817] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6497.635944] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6504.734303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6505.023371] Lustre: lustre-OST0000: new disk, initializing [ 6505.028487] Lustre: Skipped 1 previous similar message [ 6505.033293] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6505.039816] Lustre: Skipped 2 previous similar messages [ 6505.045244] Lustre: 172215:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6511.842608] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 6512.200607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6512.211357] Lustre: Skipped 1 previous similar message [ 6512.219830] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6512.280910] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6521.452811] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6526.049563] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6526.219832] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6530.898201] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6531.201346] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6538.212959] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6538.227102] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6538.234823] LustreError: Skipped 1 previous similar message [ 6538.237247] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6538.251611] Lustre: Skipped 3 previous similar messages [ 6538.269438] Lustre: Skipped 1 previous similar message [ 6539.478523] Lustre: server umount lustre-OST0000 complete [ 6549.653593] Lustre: server umount lustre-MDT0000 complete [ 6553.255165] LustreError: 170236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849031 with bad export cookie 4120435349389627336 [ 6553.265525] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6553.586561] Lustre: server umount lustre-MDT0001 complete [ 6562.049346] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 12:43:57 (1787849037) [ 6568.704582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6569.194319] LustreError: 174626:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6569.217138] LustreError: 174626:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 6573.287439] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 6581.518506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6586.046288] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 6592.790657] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6597.647073] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6604.690266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6611.578603] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 6619.447559] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6630.914297] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 12:45:06 (1787849106) [ 6638.051079] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6638.060558] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6638.076224] Lustre: Skipped 2 previous similar messages [ 6638.083729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6638.098571] Lustre: Skipped 4 previous similar messages [ 6642.037319] Lustre: server umount lustre-OST0000 complete [ 6648.291114] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6648.297841] LustreError: Skipped 1 previous similar message [ 6652.699980] Lustre: server umount lustre-MDT0000 complete [ 6656.708099] LustreError: 177401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849134 with bad export cookie 4120435349389628218 [ 6656.712727] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6656.722991] LustreError: 177401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6657.390266] Lustre: server umount lustre-MDT0001 complete [ 6668.940863] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [ 6671.530483] Key type lgssc unregistered [ 6671.794896] LNet: 178358:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6671.803107] LNetError: 178358:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6671.819911] LNet: Removed LNI 192.168.206.126@tcp [ 6672.576717] Key type .llcrypt unregistered [ 6672.580333] Key type ._llcrypt unregistered [ 6691.087192] Key type ._llcrypt registered [ 6691.090579] Key type .llcrypt registered [ 6691.244549] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 6692.502899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6692.549896] alg: No test for adler32 (adler32-zlib) [ 6693.698663] Lustre: Lustre: Build Version: 2.17.57_92_gffc8991 [ 6693.993517] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [ 6695.759194] Key type lgssc registered [ 6696.953630] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6705.216565] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6705.249039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6720.863394] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6725.983328] LustreError: 179600:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.126@tcp: failed processing log, type 4: rc = -110 [ 6731.105210] LustreError: 179600:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6731.111967] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 6731.115562] LustreError: MGC192.168.206.126@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 6731.125932] LustreError: 179600:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 6731.137441] LustreError: 179600:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 6731.142242] LustreError: 179600:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 6731.179409] Lustre: server umount lustre-OST0000 complete [ 6731.181759] LustreError: 179600:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6740.348063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6741.708876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6746.219337] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 6752.232610] LustreError: 180101:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6752.251538] LustreError: 180101:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6753.873584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6754.343076] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6758.587545] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 6765.130431] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6770.060820] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6777.159522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6777.374809] LustreError: 181914:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6777.384277] Lustre: 181914:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.206.126@tcp: invalid local config log lustre-OST0000: rc = -22 [ 6777.493807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6783.578800] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 6793.034088] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6797.281204] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6797.285864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6797.310065] Lustre: Skipped 1 previous similar message [ 6797.330873] Lustre: Skipped 2 previous similar messages [ 6802.407148] LustreError: 180100:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6802.553153] Lustre: server umount lustre-MDT0000 complete [ 6806.888721] LustreError: 180081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849284 with bad export cookie 1603238958358163717 [ 6806.906164] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6806.914219] LustreError: 180081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6807.221750] Lustre: server umount lustre-MDT0001 complete [ 6820.717964] Lustre: server umount lustre-OST0000 complete [ 6829.939639] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_hostid [ 6839.263954] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 6872.215038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6872.440630] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6872.461593] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6872.522231] Lustre: lustre-MDT0000: new disk, initializing [ 6872.631685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6872.649556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6877.421397] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 6887.010835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6887.134187] Lustre: 185441:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6887.184585] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6887.188179] Lustre: Skipped 1 previous similar message [ 6887.257291] Lustre: lustre-MDT0001: new disk, initializing [ 6887.334930] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6887.433107] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6887.442512] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6890.958542] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 6900.814307] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6906.073726] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6912.331360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6912.598738] Lustre: lustre-OST0000: new disk, initializing [ 6912.606284] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6912.613399] Lustre: 187406:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6912.762700] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6918.618502] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 6918.706826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6918.713503] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6918.868240] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6927.305974] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6931.772267] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6932.028653] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6936.104612] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6936.376924] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6939.622526] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6939.626503] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6939.641248] Lustre: Skipped 1 previous similar message [ 6939.648447] LustreError: Skipped 1 previous similar message [ 6939.649527] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6939.658350] Lustre: Skipped 1 previous similar message [ 6944.510930] Lustre: server umount lustre-OST0000 complete [ 6948.836571] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6948.844127] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6948.862866] Lustre: Skipped 1 previous similar message [ 6948.865934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6954.411240] Lustre: server umount lustre-MDT0000 complete [ 6954.987788] LustreError: 185451:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6955.007965] LustreError: 185451:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 6958.111905] LustreError: 185432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849435 with bad export cookie 1603238958358164438 [ 6958.127195] LustreError: 185432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6958.131171] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6958.449841] Lustre: server umount lustre-MDT0001 complete [ 6968.175262] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 12:50:43 (1787849443) [ 6985.366551] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [ 6988.116258] Key type lgssc unregistered [ 6988.441629] LNet: 190355:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6988.458759] LNetError: 190355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6989.478796] LNet: Removed LNI 192.168.206.126@tcp [ 6990.604117] Key type .llcrypt unregistered [ 6990.607375] Key type ._llcrypt unregistered [ 7009.103264] Key type ._llcrypt registered [ 7009.106343] Key type .llcrypt registered [ 7009.238820] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 7010.137084] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7010.164666] alg: No test for adler32 (adler32-zlib) [ 7011.250292] Lustre: Lustre: Build Version: 2.17.57_92_gffc8991 [ 7011.601903] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [ 7013.295207] Key type lgssc registered [ 7015.246020] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7022.395787] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7022.419495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7023.761800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7029.703561] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 7033.318031] LustreError: 191574:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7033.349118] LustreError: 191574:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7038.434769] LustreError: 191575:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7043.559201] LustreError: 191574:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7043.984882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7044.713905] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7051.594240] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 7061.076174] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7067.343884] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7075.002532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7075.378553] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7075.387053] LustreError: 193441:0:(llog.c:1696:llog_backup()) MGC192.168.206.126@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7075.409674] Lustre: 193441:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.206.126@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7075.626120] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7082.473400] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 7090.904698] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7095.270582] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7095.272853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7095.275910] Lustre: Skipped 1 previous similar message [ 7095.292524] Lustre: Skipped 2 previous similar messages [ 7099.168782] Lustre: server umount lustre-MDT0000 complete [ 7100.385614] LustreError: 192457:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7100.402374] LustreError: 192457:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7102.150966] LustreError: 191554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849579 with bad export cookie 8415314889376385870 [ 7102.155568] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7102.167282] LustreError: 191554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7102.524775] Lustre: server umount lustre-MDT0001 complete [ 7115.527797] Lustre: server umount lustre-OST0000 complete [ 7125.473691] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_hostid [ 7134.245317] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 7168.560397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7168.807547] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7168.850654] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7168.960810] Lustre: lustre-MDT0000: new disk, initializing [ 7169.041979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7169.063492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7173.204047] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 7182.597630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7182.744851] Lustre: 196965:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7182.804119] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7182.807484] Lustre: Skipped 1 previous similar message [ 7182.812025] LustreError: 197746:0:(llog.c:1696:llog_backup()) MGC192.168.206.126@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7182.821718] LustreError: 197746:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7182.827212] Lustre: 197746:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.126@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7182.846296] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7182.850865] Lustre: Skipped 1 previous similar message [ 7182.965300] Lustre: lustre-MDT0001: new disk, initializing [ 7183.057425] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7183.092205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7183.103567] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7187.278474] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 7195.820347] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7201.755684] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7207.527201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7207.850898] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7207.858039] Lustre: Skipped 1 previous similar message [ 7207.867946] LustreError: 198909:0:(llog.c:1696:llog_backup()) MGC192.168.206.126@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7207.875848] LustreError: 198909:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7207.884812] Lustre: 198909:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.126@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7207.907528] Lustre: 198909:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7207.968449] Lustre: lustre-OST0000: new disk, initializing [ 7207.975572] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7207.989720] Lustre: 198932:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7208.115094] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7214.510402] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 7215.173075] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7215.189067] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7215.372357] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7223.750694] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7227.960599] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7228.120228] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7232.290844] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7232.458337] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7236.067460] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7236.071830] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7236.072209] LustreError: Skipped 1 previous similar message [ 7236.073745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7236.085784] Lustre: Skipped 2 previous similar messages [ 7236.097969] Lustre: Skipped 1 previous similar message [ 7240.563522] Lustre: server umount lustre-OST0000 complete [ 7244.769579] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7244.776123] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7244.791066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7250.309863] Lustre: server umount lustre-MDT0000 complete [ 7251.425258] LustreError: 196976:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7251.458080] LustreError: 196976:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7254.093915] LustreError: 196958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849731 with bad export cookie 8415314889376386584 [ 7254.105092] LustreError: 196958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7254.107994] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7254.479663] Lustre: server umount lustre-MDT0001 complete [ 7263.239166] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 12:55:38 (1787849738) [ 7280.319782] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 7289.736929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7290.174671] LustreError: 202059:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7290.244739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7295.167155] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 7304.168139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7309.441260] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 7312.430610] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7318.304792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7324.775704] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 7333.594414] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7333.717125] Lustre: lustre-OST0001: new disk, initializing [ 7333.721737] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7333.729147] Lustre: 204367:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7333.811205] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7333.817973] Lustre: Skipped 2 previous similar messages [ 7339.471889] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 7341.161892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7341.179431] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7341.318414] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7349.848876] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7360.270167] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7362.025805] Lustre: DEBUG MARKER: ADD OST3 [ 7366.888972] loop: module loaded [ 7366.976425] loop0: detected capacity change from 0 to 204800000 [ 7367.223864] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7367.242151] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7367.265377] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7373.043908] loop0: detected capacity change from 0 to 204800000 [ 7374.085725] Lustre: DEBUG MARKER: STOP OST3 [ 7378.194639] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7378.372872] Lustre: lustre-OST0003: new disk, initializing [ 7378.381722] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7378.391365] Lustre: 206358:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7380.049392] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7380.155258] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7380.164877] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7380.192430] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7381.829539] Lustre: Failing over lustre-OST0003 [ 7381.988522] Lustre: server umount lustre-OST0003 complete [ 7382.111123] LustreError: 206546:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7383.571582] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 7385.516828] LustreError: 203546:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.206.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7385.546334] LustreError: 203546:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7385.568410] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7385.579336] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7385.593035] Lustre: Skipped 2 previous similar messages [ 7391.006536] Lustre: DEBUG MARKER: START OST3 again [ 7396.952085] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7397.233840] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7398.500178] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7398.523132] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7398.523389] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7402.864686] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 7421.980740] Lustre: Failing over lustre-OST0003 [ 7422.086186] Lustre: server umount lustre-OST0003 complete [ 7422.956933] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7422.978946] LustreError: 204447:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7422.996788] LustreError: 204447:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 9 previous similar messages [ 7442.913100] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7442.918626] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7442.929283] Lustre: Skipped 1 previous similar message [ 7442.932715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7442.939122] Lustre: Skipped 2 previous similar messages [ 7445.618452] Lustre: server umount lustre-MDT0000 complete [ 7449.434659] LustreError: 205134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849927 with bad export cookie 8415314889376387466 [ 7449.451959] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7449.451527] LustreError: 205134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7453.671680] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7453.678189] Lustre: Skipped 4 previous similar messages [ 7455.945864] Lustre: server umount lustre-MDT0001 complete [ 7460.341585] Lustre: server umount lustre-OST0000 complete [ 7464.428960] Lustre: server umount lustre-OST0001 complete [ 7470.940625] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_hostid [ 7479.265564] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 7512.939328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7513.229304] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7513.261591] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7513.322598] Lustre: lustre-MDT0000: new disk, initializing [ 7513.400190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7513.402938] Lustre: Skipped 2 previous similar messages [ 7513.423399] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7517.651229] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 7529.358370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7529.493366] Lustre: 211825:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7529.574598] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7529.582543] Lustre: Skipped 1 previous similar message [ 7529.682228] Lustre: lustre-MDT0001: new disk, initializing [ 7529.889886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7529.906701] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7534.642494] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 7543.017585] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7548.594979] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7554.540440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7554.749388] Lustre: 213793:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7556.526649] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7556.688083] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7560.262434] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 7568.686234] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7573.184113] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7573.372511] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7576.912805] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7577.088348] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7582.183812] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7582.183885] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7582.190890] Lustre: Skipped 5 previous similar messages [ 7582.194627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7582.201325] LustreError: Skipped 1 previous similar message [ 7582.226443] Lustre: Skipped 4 previous similar messages [ 7585.019364] Lustre: server umount lustre-OST0000 complete [ 7591.392664] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7595.897703] Lustre: server umount lustre-MDT0000 complete [ 7597.537976] LustreError: 212677:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7597.569734] LustreError: 212677:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 17 previous similar messages [ 7599.708146] LustreError: 211816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850077 with bad export cookie 8415314889376389762 [ 7599.712187] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7599.719673] LustreError: 211816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7600.253222] Lustre: server umount lustre-MDT0001 complete [ 7604.129606] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 13:01:19 (1787850079) [ 7620.182257] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_hostid [ 7629.448943] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 7662.802331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7663.051342] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7663.079320] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7663.084388] Lustre: Skipped 1 previous similar message [ 7663.152966] Lustre: lustre-MDT0000: new disk, initializing [ 7663.155418] Lustre: Skipped 1 previous similar message [ 7663.212504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7663.217538] Lustre: Skipped 2 previous similar messages [ 7663.228977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7663.234536] Lustre: Skipped 1 previous similar message [ 7667.817145] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 7678.539224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7678.656634] Lustre: 218413:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7679.069631] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7683.616817] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 7691.906555] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7697.018847] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7702.959703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7703.465869] Lustre: 220384:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7704.738868] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7710.040452] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 7719.211750] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7723.234415] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7723.387842] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7726.763912] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7726.933489] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7730.146754] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7730.159382] Lustre: Skipped 3 previous similar messages [ 7730.169032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7730.171088] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7730.175335] Lustre: Skipped 3 previous similar messages [ 7734.805239] Lustre: server umount lustre-OST0000 complete [ 7745.510426] LustreError: 218422:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7745.533743] LustreError: 218422:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7745.644081] Lustre: server umount lustre-MDT0000 complete [ 7749.007716] LustreError: 218408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850226 with bad export cookie 8415314889376390644 [ 7749.010776] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7749.014787] LustreError: 218408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7749.332110] Lustre: server umount lustre-MDT0001 complete [ 7757.819395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7763.463965] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 7771.552055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7775.374699] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 7782.260645] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7787.628337] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7793.731190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7799.832838] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 7806.583973] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7817.703538] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7817.721096] LustreError: Skipped 1 previous similar message [ 7817.726426] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7817.742799] Lustre: Skipped 3 previous similar messages [ 7817.752779] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7817.763907] Lustre: Skipped 2 previous similar messages [ 7822.372621] Lustre: server umount lustre-OST0000 complete [ 7840.223484] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7840.377600] Lustre: server umount lustre-MDT0000 complete [ 7843.375031] LustreError: 222585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850321 with bad export cookie 8415314889376391526 [ 7843.380621] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7843.391806] LustreError: 222585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7843.733743] Lustre: server umount lustre-MDT0001 complete [ 7944.379172] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [ 7947.383436] Key type lgssc unregistered [ 7947.780565] LNet: 226271:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7947.786253] LNetError: 226271:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7947.800192] LNet: Removed LNI 192.168.206.126@tcp [ 7948.611231] Key type .llcrypt unregistered [ 7948.618836] Key type ._llcrypt unregistered [ 7970.210901] Key type ._llcrypt registered [ 7970.212880] Key type .llcrypt registered [ 7970.307332] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_hostid [ 7983.163244] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 7984.006687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7984.022467] alg: No test for adler32 (adler32-zlib) [ 7985.118146] Lustre: Lustre: Build Version: 2.17.57_92_gffc8991 [ 7985.425317] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [ 7987.135184] Key type lgssc registered [ 7988.161776] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8019.138936] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8019.160454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8020.382609] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8020.432965] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8020.594798] Lustre: lustre-MDT0000: new disk, initializing [ 8020.693917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8020.720523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8024.709147] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 8034.730433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8034.903442] Lustre: 229602:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8035.011561] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8035.020138] Lustre: Skipped 1 previous similar message [ 8035.135199] Lustre: lustre-MDT0001: new disk, initializing [ 8035.277952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8035.321619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8035.344862] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8039.722790] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 8048.795761] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8053.957181] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8060.093647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8060.298828] Lustre: lustre-OST0000: new disk, initializing [ 8060.302308] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8060.307588] Lustre: 231572:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8060.389759] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8066.002637] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 8067.189982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8067.203577] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8067.424153] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8076.562714] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8081.026460] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8081.258570] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8086.836662] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8087.032847] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8093.155916] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8093.161042] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8093.167107] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8093.174373] Lustre: Skipped 1 previous similar message [ 8093.201337] Lustre: Skipped 1 previous similar message [ 8098.276369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8098.285437] Lustre: Skipped 1 previous similar message [ 8103.393204] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8103.399217] Lustre: Skipped 1 previous similar message [ 8103.399370] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [ 8103.577463] Lustre: server umount lustre-OST0000 complete [ 8106.977200] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8106.987637] LustreError: Skipped 1 previous similar message [ 8106.991616] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8107.003480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8108.528884] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8113.632114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8113.638618] Lustre: Skipped 2 previous similar messages [ 8121.311218] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8121.640462] Lustre: server umount lustre-MDT0000 complete [ 8125.994067] LustreError: 230396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850603 with bad export cookie 8505052947303261668 [ 8126.012866] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8126.015634] LustreError: 230396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8126.411976] Lustre: server umount lustre-MDT0001 complete [ 8143.321353] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 8152.931954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8153.319040] LustreError: 234508:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8153.345904] LustreError: 234508:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 8153.383612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8157.796441] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 8158.694706] LustreError: 234509:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8163.816667] LustreError: 234508:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8166.298457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8166.791810] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8171.457530] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 8174.232943] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8180.726156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8181.068949] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8187.163892] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 8196.077604] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8196.332143] Lustre: lustre-OST0001: new disk, initializing [ 8196.336348] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8196.343768] Lustre: 236813:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8203.376679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8203.388852] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8203.388931] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 8203.502406] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8214.892264] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8224.067811] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8236.660295] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 13:11:52 (1787850712) [ 8239.586748] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8239.588834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8239.596289] Lustre: Skipped 2 previous similar messages [ 8239.603870] Lustre: Skipped 6 previous similar messages [ 8252.895155] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8253.423377] Lustre: server umount lustre-MDT0000 complete [ 8254.946557] LustreError: 234508:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8254.969455] LustreError: 234508:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 8257.772922] LustreError: 234489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850735 with bad export cookie 8505052947303262550 [ 8257.780166] LustreError: 234489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8257.785258] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8258.061699] Lustre: server umount lustre-MDT0001 complete [ 8271.516596] Lustre: server umount lustre-OST0000 complete [ 8284.547691] Lustre: server umount lustre-OST0001 complete [ 8291.082914] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_hostid [ 8300.167531] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 8336.332941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8336.480751] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8336.504689] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8336.556952] Lustre: lustre-MDT0000: new disk, initializing [ 8336.607592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8336.616451] Lustre: Skipped 1 previous similar message [ 8336.632613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8340.145819] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 8351.357296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8351.533964] Lustre: 240761:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8351.593432] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8351.597215] Lustre: Skipped 1 previous similar message [ 8351.662214] Lustre: lustre-MDT0001: new disk, initializing [ 8351.877911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8351.884772] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8359.960809] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 8374.434688] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8381.700840] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8388.560181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8388.909834] Lustre: lustre-OST0000: new disk, initializing [ 8388.912608] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8388.917374] Lustre: 242735:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8388.985560] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8388.987793] Lustre: Skipped 1 previous similar message [ 8390.420809] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8390.436646] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8390.585944] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8395.779085] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 8405.772979] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8410.744889] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8410.971768] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8415.888682] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8416.071878] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8421.349066] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8421.350832] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8421.358688] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8421.363484] Lustre: Skipped 2 previous similar messages [ 8421.366593] Lustre: Skipped 8 previous similar messages [ 8432.096915] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8432.214522] Lustre: server umount lustre-OST0000 complete [ 8436.714306] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8442.108312] Lustre: server umount lustre-MDT0000 complete [ 8445.807933] LustreError: 241558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850923 with bad export cookie 8505052947303264685 [ 8445.813265] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8445.829519] LustreError: 241558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 8446.324703] Lustre: server umount lustre-MDT0001 complete [ 8454.609550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8455.030234] LustreError: 244952:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8455.101788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8459.852459] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 8464.356602] LustreError: 244953:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8464.375597] LustreError: 244953:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8467.933308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8472.515877] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 8479.983384] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8485.007861] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8491.225152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8497.756761] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 8504.729756] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8651.744962] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8651.749973] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8651.762665] Lustre: Skipped 1 previous similar message [ 8651.772051] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8651.777049] Lustre: Skipped 9 previous similar messages [ 8657.661833] Lustre: server umount lustre-OST0000 complete [ 8663.009538] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8663.015391] LustreError: Skipped 1 previous similar message [ 8667.443351] Lustre: server umount lustre-MDT0000 complete [ 8669.153631] LustreError: 245680:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8669.170867] LustreError: 245680:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8670.678227] LustreError: 244935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851148 with bad export cookie 8505052947303265567 [ 8670.685863] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8670.688681] LustreError: 244935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8671.008167] Lustre: server umount lustre-MDT0001 complete [ 8772.558827] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [ 8775.234361] Key type lgssc unregistered [ 8775.682738] LNet: 248563:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8775.687366] LNetError: 248563:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8775.702679] LNet: Removed LNI 192.168.206.126@tcp [ 8776.791134] Key type .llcrypt unregistered [ 8776.792919] Key type ._llcrypt unregistered [ 8805.627832] Key type ._llcrypt registered [ 8805.630286] Key type .llcrypt registered [ 8805.727824] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_hostid [ 8822.855678] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 8824.476029] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8824.572701] alg: No test for adler32 (adler32-zlib) [ 8825.580674] Lustre: Lustre: Build Version: 2.17.57_92_gffc8991 [ 8825.768780] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [ 8827.431184] Key type lgssc registered [ 8828.795300] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8869.435317] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 8878.624864] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8878.653978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8879.871183] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8879.894907] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8879.973227] Lustre: lustre-MDT0000: new disk, initializing [ 8880.023505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8880.041540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8884.753552] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 8893.521974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8893.638830] Lustre: 252633:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8893.690563] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8893.694743] Lustre: Skipped 1 previous similar message [ 8893.763296] Lustre: lustre-MDT0001: new disk, initializing [ 8893.863089] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8893.903985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8893.917030] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8897.846652] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 8902.110524] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8907.387787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8907.635726] Lustre: lustre-OST0000: new disk, initializing [ 8907.639184] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8907.645089] Lustre: 254265:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8907.720630] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8914.366378] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 8914.538887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8914.551529] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8914.700900] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8924.148503] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8924.348495] Lustre: lustre-OST0001: new disk, initializing [ 8924.353249] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8924.366656] Lustre: 255136:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8924.460964] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8930.981575] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 8931.940100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8931.956220] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8932.090729] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8940.663306] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8951.460345] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8963.050920] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8963.057488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8963.058369] Lustre: Skipped 2 previous similar messages [ 8963.068073] Lustre: Skipped 3 previous similar messages [ 8963.808604] Lustre: server umount lustre-MDT0000 complete [ 8967.849131] LustreError: 252625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851445 with bad export cookie 8615691335526334294 [ 8967.867594] LustreError: 252625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8967.869625] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8968.160730] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8968.169122] Lustre: Skipped 1 previous similar message [ 8968.264392] Lustre: server umount lustre-MDT0001 complete [ 8974.575239] Lustre: server umount lustre-OST0000 complete [ 8978.162602] Lustre: server umount lustre-OST0001 complete [ 8990.703772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9001.279255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9010.563477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9019.328301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9039.169757] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 9050.432679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9050.460942] Lustre: lustre-MDT0000: reset Object Index mappings [ 9050.844063] LustreError: 260145:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9050.858190] LustreError: 260145:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 9050.929991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9056.028776] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 9056.226219] LustreError: 260144:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9060.322908] LustreError: 260145:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9065.446434] LustreError: 260144:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9065.523172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9065.537072] Lustre: lustre-MDT0001: reset Object Index mappings [ 9065.803577] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9070.742026] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 9073.998613] Lustre: 261285:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9080.268724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9086.617441] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 9090.790599] LustreError: 261638:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9095.857179] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9095.925827] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9096.159035] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9096.172177] Lustre: Skipped 1 previous similar message [ 9101.308152] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9102.302526] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 9108.928351] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9112.114262] Lustre: 263155:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9116.644741] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9116.649377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9116.649930] Lustre: Skipped 4 previous similar messages [ 9116.664752] Lustre: Skipped 3 previous similar messages [ 9118.541588] Lustre: server umount lustre-MDT0000 complete [ 9121.764177] LustreError: 260139:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9121.784178] LustreError: 260139:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 9122.813518] LustreError: 260126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851600 with bad export cookie 8615691335526341091 [ 9122.820397] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9122.824435] LustreError: 260126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9123.289793] Lustre: server umount lustre-MDT0001 complete [ 9137.808036] Lustre: server umount lustre-OST0000 complete [ 9150.529297] Lustre: server umount lustre-OST0001 complete [ 9168.168452] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 13:27:23 (1787851643) [ 9183.369707] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_hostid [ 9194.900232] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 9238.380613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9238.924436] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9238.957327] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9239.180840] Lustre: lustre-MDT0000: new disk, initializing [ 9239.352206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9239.388139] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9245.635686] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 9257.660624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9257.749215] Lustre: 267411:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9257.830280] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9257.837854] Lustre: Skipped 1 previous similar message [ 9257.917542] Lustre: lustre-MDT0001: new disk, initializing [ 9258.041156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9258.055991] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9263.883172] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 9272.856070] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9277.536558] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9282.958178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9283.374520] Lustre: lustre-OST0000: new disk, initializing [ 9283.380987] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9283.398669] Lustre: 269380:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9285.076018] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9285.089687] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9285.274245] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9290.460784] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 9300.048250] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9305.416595] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9305.594927] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9310.674626] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9310.904826] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9315.821810] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9315.824719] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9315.837139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9315.851765] Lustre: Skipped 1 previous similar message [ 9319.068849] Lustre: server umount lustre-OST0000 complete [ 9324.512349] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9324.528522] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9324.543380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9324.552584] Lustre: Skipped 1 previous similar message [ 9331.169949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9331.177967] Lustre: Skipped 3 previous similar messages [ 9337.311147] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9337.603945] Lustre: server umount lustre-MDT0000 complete [ 9340.386503] LustreError: 267422:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9340.419970] LustreError: 267422:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9341.564346] LustreError: 267403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851819 with bad export cookie 8615691335526343184 [ 9341.575544] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9341.580818] LustreError: 267403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9342.109331] Lustre: server umount lustre-MDT0001 complete [ 9361.338622] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 9371.202577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9371.882022] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9371.887801] Lustre: Skipped 2 previous similar messages [ 9377.267441] LustreError: 272315:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9377.300114] LustreError: 272315:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9377.928844] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 9386.971243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9391.511486] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 9394.434310] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9400.717083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9408.349162] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 9416.804039] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9417.041625] Lustre: lustre-OST0001: new disk, initializing [ 9417.045814] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9417.058791] Lustre: 274621:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9418.412161] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9418.422461] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9418.547988] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9424.698621] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 9433.598302] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9436.908737] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9441.928091] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 9442.013991] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 9449.449502] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9449.453804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9449.470304] Lustre: Skipped 4 previous similar messages [ 9449.480074] Lustre: Skipped 5 previous similar messages [ 9455.357728] Lustre: server umount lustre-MDT0000 complete [ 9458.662544] LustreError: 272297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851936 with bad export cookie 8615691335526344059 [ 9458.677860] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9458.685073] LustreError: 272297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9458.988793] Lustre: server umount lustre-MDT0001 complete [ 9473.000761] Lustre: server umount lustre-OST0000 complete [ 9476.063215] Lustre: 250142:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787851937/real 1787851937] req@ffff90d24b23c3c0 x1874697968124800/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787851953 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9476.101159] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9477.019653] Lustre: server umount lustre-OST0001 complete [ 9497.187808] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 9508.940143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9509.513627] LustreError: 277681:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9509.529138] LustreError: 277681:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 9509.621529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9509.638514] Lustre: Skipped 3 previous similar messages [ 9515.218615] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 9524.570454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9529.766810] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 9532.931092] Lustre: 278824:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9539.286482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9545.499648] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 9555.968749] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 9556.104510] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9561.626177] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 9563.939170] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 9571.819830] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9574.979552] Lustre: 280699:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9579.815640] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 9580.030237] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 9589.636194] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 13:34:25 (1787852065) [ 9607.136847] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9607.141900] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9607.157671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9607.169674] Lustre: Skipped 4 previous similar messages [ 9610.410206] Lustre: server umount lustre-MDT0000 complete [ 9613.889816] LustreError: 278425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852091 with bad export cookie 8615691335526350632 [ 9613.899025] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9613.900352] LustreError: 278425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9620.260939] Lustre: server umount lustre-MDT0001 complete [ 9631.479775] Lustre: server umount lustre-OST0000 complete [ 9642.910968] Lustre: server umount lustre-OST0001 complete [ 9650.834507] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_hostid [ 9662.235791] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 9723.598300] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [ 9733.659468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9733.883300] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9733.942218] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9734.063446] Lustre: lustre-MDT0000: new disk, initializing [ 9734.206639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9738.461492] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 9748.565850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9748.718822] Lustre: 284941:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9749.003968] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9753.893912] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 9765.726431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9766.245530] Lustre: 286576:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9766.309203] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9766.316127] Lustre: Skipped 5 previous similar messages [ 9767.904672] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9774.351360] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 9785.712256] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9785.919598] Lustre: 287446:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9787.836594] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9787.849381] Lustre: Skipped 1 previous similar message [ 9787.978263] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9793.173617] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 9804.176433] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9807.524690] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9807.529683] Lustre: Skipped 1 previous similar message [ 9822.869098] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 13:38:18 (1787852298) [ 9825.763039] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9825.777566] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9825.785694] Lustre: Skipped 5 previous similar messages [ 9825.789293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9825.791814] Lustre: Skipped 7 previous similar messages [ 9827.055514] Lustre: server umount lustre-MDT0000 complete [ 9828.833377] LustreError: 285819:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9828.851835] LustreError: 285819:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 17 previous similar messages [ 9830.142492] LustreError: 285738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852307 with bad export cookie 8615691335526358304 [ 9830.149906] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9830.160836] LustreError: 285738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9830.400781] Lustre: server umount lustre-MDT0001 complete [ 9844.428186] Lustre: server umount lustre-OST0000 complete [ 9857.431444] Lustre: server umount lustre-OST0001 complete [ 9865.624830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9870.854514] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 9878.822681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9883.500746] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 9891.862766] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9897.884377] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9906.066333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9915.202601] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 9924.948074] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9945.014750] loop1: detected capacity change from 0 to 204800000 [ 9945.220688] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9945.226701] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9945.237286] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9949.833289] loop1: detected capacity change from 0 to 204800000 [ 9954.940846] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9955.041857] LustreError: 289681:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs4473: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 9955.059515] LustreError: 289681:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs4473 [ 9955.065914] LustreError: 289681:0:(mgs_handler.c:517:mgs_target_reg()) fs4473-MDT0001: Error checking target index: rc = -13 [ 9955.082526] LustreError: fs4473-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 9955.091083] LustreError: 293412:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 9955.104402] LustreError: 293412:0:(tgt_mount.c:2129:server_put_super()) no obd fs4473-MDT0001 [ 9955.114733] LustreError: 293412:0:(tgt_mount.c:116:server_deregister_mount()) fs4473-MDT0001 not registered [ 9955.162746] Lustre: server umount fs4473-MDT0001 complete [ 9955.169738] LustreError: 293412:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 9964.167324] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9964.310261] Lustre: Setting parameter fs4473-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs4473-MDT0001 [ 9964.389925] Lustre: srv-fs4473-MDT0001: No data found on store. Initialize space. [ 9964.398959] Lustre: Skipped 4 previous similar messages [ 9964.539877] Lustre: fs4473-MDT0001: new disk, initializing [ 9964.546865] Lustre: Skipped 3 previous similar messages [ 9964.701625] Lustre: srv-fs4473-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 9964.724900] Lustre: 293881:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs4473-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9964.751328] LustreError: 293881:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs4473-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9969.642716] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [ 9976.737690] Lustre: server umount fs4473-MDT0001 complete [ 9990.691882] loop1: detected capacity change from 0 to 204800000 [ 9991.669508] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [ 9997.350506] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9998.227428] Lustre: 295448:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs4473-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9998.245462] LustreError: 295448:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs4473-MDT0001-osd: get update log duration 1, retries 0, failed: rc = -115 [10002.825093] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [10007.042626] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10008.298825] Lustre: server umount fs4473-MDT0001 complete [10022.368343] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10022.379151] Lustre: Skipped 4 previous similar messages [10022.391634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10022.399251] Lustre: Skipped 2 previous similar messages [10024.488493] Lustre: server umount lustre-MDT0000 complete [10036.244027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10036.472367] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10042.371140] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10042.920380] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [10053.132680] loop1: detected capacity change from 0 to 204800000 [10057.682737] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10058.268537] Lustre: 297898:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs4473-MDT0001-mdtlov: can't create new llog: rc = -115 [10058.283935] LustreError: 297898:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs4473-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10061.487534] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [10065.782971] Lustre: server umount fs4473-MDT0001 complete [10087.908900] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10088.421918] LustreError: 291524:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10088.448265] LustreError: 291524:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 108 previous similar messages [10093.321992] Lustre: server umount lustre-OST0000 complete [10105.981405] Lustre: server umount lustre-MDT0000 complete [10109.945065] LustreError: 289674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852587 with bad export cookie 8615691335526360992 [10109.951082] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10109.951600] LustreError: 289674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10110.314813] Lustre: server umount lustre-MDT0001 complete [10122.268829] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [10125.221192] Key type lgssc unregistered [10125.485227] LNet: 300243:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10125.489865] LNetError: 300243:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10125.523496] LNet: Removed LNI 192.168.206.126@tcp [10126.265227] Key type .llcrypt unregistered [10126.267813] Key type ._llcrypt unregistered [10145.927411] Key type ._llcrypt registered [10145.936804] Key type .llcrypt registered [10146.057182] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [10146.848488] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10146.932567] alg: No test for adler32 (adler32-zlib) [10148.273372] Lustre: Lustre: Build Version: 2.17.57_92_gffc8991 [10148.783342] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [10150.568102] Key type lgssc registered [10153.022447] Lustre: Echo OBD driver; http://www.lustre.org/ [10162.917322] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10162.929936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10164.257041] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10169.739921] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [10173.410555] LustreError: 301517:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10173.436514] LustreError: 301517:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10177.800959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10178.145149] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10182.198680] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [10190.915262] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10196.259277] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10203.979993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10204.387455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10208.498290] LustreError: 303346:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10211.032502] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [10213.865880] LustreError: 303552:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10219.002852] LustreError: 303346:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10219.163818] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10223.089924] LustreError: 303347:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10223.122513] LustreError: 303347:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10231.187073] LustreError: 303347:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10231.222884] LustreError: 303347:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [10242.769648] loop1: detected capacity change from 0 to 204800000 [10242.948347] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10242.977635] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10243.002380] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10248.678603] LustreError: 303345:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10248.705744] LustreError: 303345:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 10 previous similar messages [10248.728402] loop1: detected capacity change from 0 to 204800000 [10254.230612] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10254.389559] Lustre: fs3746-OST0000: new disk, initializing [10254.394671] Lustre: srv-fs3746-OST0000: No data found on store. Initialize space. [10254.400831] Lustre: 305250:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10254.491636] Lustre: fs3746-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10261.083113] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [10269.210001] Lustre: server umount fs3746-OST0000 complete [10279.812455] loop1: detected capacity change from 0 to 204800000 [10282.387445] LustreError: 303552:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10282.406865] LustreError: 303552:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 18 previous similar messages [10285.756048] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10285.991974] Lustre: fs3746-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10289.637702] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10290.889752] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [10298.138234] loop2: detected capacity change from 0 to 204800000 [10298.272460] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10298.281487] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10298.287492] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10302.538338] loop2: detected capacity change from 0 to 204800000 [10305.011175] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10308.491730] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10308.556444] LustreError: fs3746-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10308.563084] LustreError: fs3746-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10308.567572] LustreError: 308289:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10308.571665] LustreError: 308289:0:(tgt_mount.c:2129:server_put_super()) no obd fs3746-OST0001 [10308.575584] LustreError: 308289:0:(tgt_mount.c:116:server_deregister_mount()) fs3746-OST0001 not registered [10308.600768] Lustre: server umount fs3746-OST0001 complete [10308.602227] LustreError: 308289:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10319.321489] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10319.524603] Lustre: fs3746-OST0001: new disk, initializing [10319.528992] Lustre: srv-fs3746-OST0001: No data found on store. Initialize space. [10319.538509] Lustre: 308841:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10319.621699] Lustre: fs3746-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10325.669155] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [10333.463524] Lustre: server umount fs3746-OST0000 complete [10342.337471] Lustre: server umount fs3746-OST0001 complete [10348.961805] LustreError: 303552:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10348.999636] LustreError: 303552:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 38 previous similar messages [10356.273375] loop1: detected capacity change from 0 to 204800000 [10362.697276] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10362.891312] Lustre: fs3746-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10369.812739] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [10376.227339] loop2: detected capacity change from 0 to 204800000 [10381.504282] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10388.633406] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [10396.964098] Lustre: server umount fs3746-OST0000 complete [10405.540592] Lustre: server umount fs3746-OST0001 complete [10417.649214] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10417.675794] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10417.679097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10417.697737] Lustre: Skipped 1 previous similar message [10422.759869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10422.774783] Lustre: Skipped 1 previous similar message [10423.309720] Lustre: server umount lustre-OST0000 complete [10427.877401] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10427.878417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10427.891246] Lustre: Skipped 2 previous similar messages [10427.899925] Lustre: Skipped 1 previous similar message [10432.994633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10433.005368] Lustre: Skipped 1 previous similar message [10433.525750] Lustre: server umount lustre-MDT0000 complete [10436.575403] LustreError: 301499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852914 with bad export cookie 17360144514096925523 [10436.581412] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10436.584854] LustreError: 301499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10436.831895] Lustre: server umount lustre-MDT0001 complete [10445.797933] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [10448.456206] Key type lgssc unregistered [10448.680682] LNet: 314769:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10448.686209] LNetError: 314769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10448.698061] LNet: Removed LNI 192.168.206.126@tcp [10449.597207] Key type .llcrypt unregistered [10449.600263] Key type ._llcrypt unregistered [10479.451502] Key type ._llcrypt registered [10479.453708] Key type .llcrypt registered [10479.532455] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_hostid [10493.419698] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [10494.319029] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10494.440719] alg: No test for adler32 (adler32-zlib) [10495.471435] Lustre: Lustre: Build Version: 2.17.57_92_gffc8991 [10495.773592] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [10497.463170] Key type lgssc registered [10498.669693] Lustre: Echo OBD driver; http://www.lustre.org/ [10531.740798] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10531.772799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10532.921282] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10532.957424] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10533.045170] Lustre: lustre-MDT0000: new disk, initializing [10533.134907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10533.150387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10537.385286] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [10546.262443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10546.402644] Lustre: 318631:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10546.489328] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10546.492268] Lustre: Skipped 1 previous similar message [10546.537482] Lustre: lustre-MDT0001: new disk, initializing [10546.610112] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10546.659611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10546.669308] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10550.278529] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [10557.601764] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10561.592081] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10567.682709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10568.239411] Lustre: lustre-OST0000: new disk, initializing [10568.242542] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10568.264541] Lustre: 320597:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10568.396881] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10574.558605] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [10575.096074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10575.117690] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10575.346922] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10587.138888] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10589.568720] Lustre: Mounted lustre-client - version 2.17.57_92_gffc8991 [10590.448891] Lustre: Unmounted lustre-client [10599.906715] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10599.912057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10599.918821] Lustre: Skipped 1 previous similar message [10604.700724] Lustre: server umount lustre-OST0000 complete [10608.096429] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10608.101380] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10608.108923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10608.111569] Lustre: Skipped 1 previous similar message [10610.144420] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10610.147819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10610.159709] Lustre: Skipped 1 previous similar message [10616.270907] Lustre: server umount lustre-MDT0000 complete [10619.296777] LustreError: 318623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853097 with bad export cookie 14005563707289041869 [10619.304699] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10619.308972] LustreError: 318623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10619.575487] Lustre: server umount lustre-MDT0001 complete [10629.396788] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [10631.992216] Key type lgssc unregistered [10651.743489] LNet: 322764:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10651.766995] LNetError: 322764:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10720.421919] LNet: Removed LNI 192.168.206.126@tcp [10721.282085] Key type .llcrypt unregistered [10721.283489] Key type ._llcrypt unregistered [10738.509355] Key type ._llcrypt registered [10738.514848] Key type .llcrypt registered [10738.641309] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_hostid [10753.709844] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [10755.054652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10755.158090] alg: No test for adler32 (adler32-zlib) [10756.259543] Lustre: Lustre: Build Version: 2.17.57_92_gffc8991 [10756.587205] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [10758.296640] Key type lgssc registered [10759.761339] Lustre: Echo OBD driver; http://www.lustre.org/ [10788.510307] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 13:54:24 (1787853264) [10793.895455] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10793.921036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10795.119614] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10795.159722] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10795.269780] Lustre: lustre-MDT0000: new disk, initializing [10795.358055] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10795.372169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10799.794744] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [10809.526722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10809.651791] Lustre: 325988:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10809.712575] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10809.716952] Lustre: Skipped 1 previous similar message [10809.795539] Lustre: lustre-MDT0001: new disk, initializing [10809.898687] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10809.950467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10809.961914] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10813.149914] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [10821.739907] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10827.532454] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10833.999576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10834.481360] Lustre: lustre-OST0000: new disk, initializing [10834.485290] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10834.496201] Lustre: 327956:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10834.645562] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10840.837366] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [10842.697213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10842.704319] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10842.866843] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10850.141661] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10868.711849] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10868.715560] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10868.718656] LustreError: Skipped 1 previous similar message [10868.720280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10868.735751] Lustre: Skipped 1 previous similar message [10870.361915] Lustre: server umount lustre-OST0000 complete [10873.830941] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10873.835352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10873.848985] Lustre: Skipped 1 previous similar message [10873.858059] Lustre: Skipped 2 previous similar messages [10878.944354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10878.953701] Lustre: Skipped 1 previous similar message [10884.064200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10884.072467] Lustre: Skipped 1 previous similar message [10888.159861] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10888.331720] Lustre: server umount lustre-MDT0000 complete [10891.742940] LustreError: 325978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853369 with bad export cookie 15048663917657772418 [10891.752776] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10891.753508] LustreError: 325978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10892.055479] Lustre: server umount lustre-MDT0001 complete [10900.702680] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [10903.313429] Key type lgssc unregistered [10903.647240] LNet: 329866:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10903.655723] LNetError: 329866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10903.672506] LNet: Removed LNI 192.168.206.126@tcp [10904.391357] Key type .llcrypt unregistered [10904.393848] Key type ._llcrypt unregistered [10924.227594] Key type ._llcrypt registered [10924.229249] Key type .llcrypt registered [10924.345231] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [10925.304088] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10925.338068] alg: No test for adler32 (adler32-zlib) [10926.487749] Lustre: Lustre: Build Version: 2.17.57_92_gffc8991 [10926.747074] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [10928.455155] Key type lgssc registered [10929.722920] Lustre: Echo OBD driver; http://www.lustre.org/ [10936.832424] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10936.845292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10938.161543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10941.426220] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [10947.555346] LustreError: 331140:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10947.574788] LustreError: 331140:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10947.985783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10948.435545] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10953.561311] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [10960.638122] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10965.877901] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10972.047992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10972.410265] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10977.440710] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [10984.172821] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10992.906808] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10994.302223] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10998.242771] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10998.250285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10998.261526] Lustre: Skipped 2 previous similar messages [10998.273773] Lustre: Skipped 2 previous similar messages [11002.275864] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.26@tcp (stopping) [11003.363810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11003.371741] Lustre: Skipped 2 previous similar messages [11004.293705] Lustre: server umount lustre-MDT0000 complete [11007.374814] LustreError: 333580:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11008.250531] LustreError: 331122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853486 with bad export cookie 15621998312996119189 [11008.265018] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11008.482964] LustreError: 332382:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11008.506192] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11008.562527] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11012.503330] LustreError: 333580:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11013.600474] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11013.612200] Lustre: Skipped 1 previous similar message [11014.589575] Lustre: server umount lustre-MDT0001 complete [11018.934318] Lustre: server umount lustre-OST0000 complete [11027.204088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11031.980395] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [11034.770423] Lustre: 335320:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.126@tcp [11036.998918] Lustre: server umount MGS complete [11045.312658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11045.716071] LustreError: 335892:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11045.732771] LustreError: 335892:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [11045.803757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11050.056135] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [11056.102785] LustreError: 335892:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11056.118234] LustreError: 335892:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11058.092757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11058.541844] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11062.635423] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [11070.163537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11070.491441] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11076.306995] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [11083.188931] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11084.257136] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11089.442858] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11091.097544] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11097.058501] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11097.069988] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11097.083785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11097.458434] Lustre: server umount lustre-OST0000 complete [11104.223978] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11104.238744] Lustre: Skipped 1 previous similar message [11107.124119] Lustre: server umount lustre-MDT0000 complete [11109.347941] LustreError: 335892:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11109.369704] LustreError: 335892:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11110.500232] LustreError: 335874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853588 with bad export cookie 15621998312996120022 [11110.513359] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11110.514130] LustreError: 335874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [11121.220174] Lustre: server umount lustre-MDT0001 complete [11131.085625] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [11133.970715] Key type lgssc unregistered [11134.360574] LNet: 339331:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11134.373911] LNetError: 339331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11134.400879] LNet: Removed LNI 192.168.206.126@tcp [11135.358735] Key type .llcrypt unregistered [11135.361164] Key type ._llcrypt unregistered [11151.740781] Key type ._llcrypt registered [11151.743757] Key type .llcrypt registered [11151.837141] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_hostid [11166.237589] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [11167.213801] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11167.237080] alg: No test for adler32 (adler32-zlib) [11168.510497] Lustre: Lustre: Build Version: 2.17.57_92_gffc8991 [11168.777756] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [11170.511424] Key type lgssc registered [11171.734652] Lustre: Echo OBD driver; http://www.lustre.org/ [11205.023752] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11205.062963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11206.394566] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11206.457928] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11206.530426] Lustre: lustre-MDT0000: new disk, initializing [11206.630238] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11206.642621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11210.898486] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [11221.112139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11221.240710] Lustre: 342504:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11221.295848] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11221.299319] Lustre: Skipped 1 previous similar message [11221.358677] Lustre: lustre-MDT0001: new disk, initializing [11221.445787] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11221.478078] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11221.500694] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11225.934103] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [11234.278218] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11239.704847] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11245.772668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11246.027693] Lustre: lustre-OST0000: new disk, initializing [11246.030742] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11246.035142] Lustre: 344472:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11246.114980] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11251.727527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11251.738822] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11251.848034] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [11251.884696] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11259.971500] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11263.662934] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11263.872703] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11267.701660] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11267.862494] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11272.162955] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11272.175435] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11272.184540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11272.201675] Lustre: Skipped 1 previous similar message [11272.218496] Lustre: Skipped 1 previous similar message [11275.525886] Lustre: server umount lustre-OST0000 complete [11282.406100] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11282.409638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11282.426016] Lustre: Skipped 1 previous similar message [11285.261026] Lustre: server umount lustre-MDT0000 complete [11288.247138] LustreError: 342496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853766 with bad export cookie 14012768985909665029 [11288.248779] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11288.257706] LustreError: 342496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11288.568860] Lustre: server umount lustre-MDT0001 complete [11295.709240] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 14:02:51 (1787853771) [11309.066358] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_hostid [11317.831611] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [11349.001708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11349.265844] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11349.287695] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11349.347177] Lustre: lustre-MDT0000: new disk, initializing [11349.425738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11349.446307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11353.419269] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [11363.458779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11363.581127] Lustre: 349225:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11363.625588] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11363.628788] Lustre: Skipped 1 previous similar message [11363.690350] Lustre: lustre-MDT0001: new disk, initializing [11363.798103] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11363.862925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11363.880191] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11368.637517] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [11378.566318] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11383.735541] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11390.647578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11390.928269] Lustre: lustre-OST0000: new disk, initializing [11390.931676] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11390.937133] Lustre: 351193:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11391.048312] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11396.624693] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [11400.762025] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11400.776818] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11400.899986] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11405.146796] Lustre: DEBUG MARKER: oleg626-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11442.144242] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11442.154163] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11442.171812] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11442.179658] Lustre: Skipped 2 previous similar messages [11443.936188] Lustre: server umount lustre-OST0000 complete [11447.271118] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11447.279657] Lustre: Skipped 1 previous similar message [11447.284833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11447.290058] Lustre: Skipped 1 previous similar message [11452.384854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11452.390498] Lustre: Skipped 1 previous similar message [11453.495528] Lustre: server umount lustre-MDT0000 complete [11457.386854] LustreError: 349220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853935 with bad export cookie 14012768985909665911 [11457.397832] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11457.405328] LustreError: 349220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11457.700768] Lustre: server umount lustre-MDT0001 complete [11467.490607] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [11470.496493] Key type lgssc unregistered [11470.871618] LNet: 353369:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11470.886524] LNetError: 353369:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11470.939123] LNet: Removed LNI 192.168.206.126@tcp [11471.960611] Key type .llcrypt unregistered [11471.964326] Key type ._llcrypt unregistered [11558.116241] Key type ._llcrypt registered [11558.118748] Key type .llcrypt registered [11558.229452] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [11558.534411] Key type .llcrypt unregistered [11558.539380] Key type ._llcrypt unregistered [11639.553260] Key type ._llcrypt registered [11639.559103] Key type .llcrypt registered [11639.740855] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [11640.132458] Key type .llcrypt unregistered [11640.134220] Key type ._llcrypt unregistered [11669.692480] Key type ._llcrypt registered [11669.696814] Key type .llcrypt registered [11669.809770] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [11670.272894] Key type .llcrypt unregistered [11670.274484] Key type ._llcrypt unregistered [11701.668768] Key type ._llcrypt registered [11701.674828] Key type .llcrypt registered [11701.838048] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [11702.186456] Key type .llcrypt unregistered [11702.192505] Key type ._llcrypt unregistered [11777.319664] Key type ._llcrypt registered [11777.324197] Key type .llcrypt registered [11777.457634] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [11777.746310] Key type .llcrypt unregistered [11777.750526] Key type ._llcrypt unregistered [11805.070920] Key type ._llcrypt registered [11805.072314] Key type .llcrypt registered [11805.167525] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [11805.459627] Key type .llcrypt unregistered [11805.462536] Key type ._llcrypt unregistered [11832.424585] Key type ._llcrypt registered [11832.426920] Key type .llcrypt registered [11832.570200] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [11832.954163] Key type .llcrypt unregistered [11832.955924] Key type ._llcrypt unregistered [11862.362960] Key type ._llcrypt registered [11862.367381] Key type .llcrypt registered [11862.507405] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [11862.952596] Key type .llcrypt unregistered [11862.956310] Key type ._llcrypt unregistered [11894.690029] Key type ._llcrypt registered [11894.694397] Key type .llcrypt registered [11894.809983] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [11895.174210] Key type .llcrypt unregistered [11895.177964] Key type ._llcrypt unregistered [11923.557216] Key type ._llcrypt registered [11923.569482] Key type .llcrypt registered [11923.738380] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [11924.158363] Key type .llcrypt unregistered [11924.163541] Key type ._llcrypt unregistered [11979.765538] Key type ._llcrypt registered [11979.768367] Key type .llcrypt registered [11979.872080] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [11980.177282] Key type .llcrypt unregistered [11980.180756] Key type ._llcrypt unregistered [12009.551763] Key type ._llcrypt registered [12009.552943] Key type .llcrypt registered [12009.648771] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [12009.879303] Key type .llcrypt unregistered [12009.881164] Key type ._llcrypt unregistered [12038.244809] Key type ._llcrypt registered [12038.248181] Key type .llcrypt registered [12038.386955] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [12038.715760] Key type .llcrypt unregistered [12038.721347] Key type ._llcrypt unregistered [12087.251304] Key type ._llcrypt registered [12087.253681] Key type .llcrypt registered [12087.364996] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_hostid [12100.909863] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [12101.667382] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12101.748059] alg: No test for adler32 (adler32-zlib) [12102.942449] Lustre: Lustre: Build Version: 2.17.57_92_gffc8991 [12103.183853] LNet: Added LNI 192.168.206.126@tcp [8/256/0/180] [12104.871167] Key type lgssc registered [12105.866537] Lustre: Echo OBD driver; http://www.lustre.org/ [12140.987502] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12141.017965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12142.267597] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [12142.326987] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [12142.455158] Lustre: tst250fs-MDT0000: new disk, initializing [12142.541612] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12142.558192] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12146.430482] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [12156.003442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12156.118776] Lustre: 373677:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12156.165413] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [12156.169360] Lustre: Skipped 1 previous similar message [12156.230677] Lustre: tst250fs-MDT0001: new disk, initializing [12156.307159] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12156.357204] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12156.370069] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12160.720705] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [12165.751657] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12172.900736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12173.197723] Lustre: tst250fs-OST0000: new disk, initializing [12173.205346] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [12173.218593] Lustre: 375310:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12173.314986] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12179.008615] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12179.018893] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12179.126665] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12180.066897] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [12189.827193] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12190.066771] Lustre: tst250fs-OST0001: new disk, initializing [12190.072435] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [12190.088290] Lustre: 376182:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12190.192683] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12196.573216] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_default_debug -1 all [12197.011130] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12197.016174] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12197.127778] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12202.415607] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [12205.100717] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [12205.107643] Lustre: Skipped 2 previous similar messages [12228.071587] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12228.085659] Lustre: Skipped 2 previous similar messages [12228.098080] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12228.101473] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12228.109824] Lustre: Skipped 3 previous similar messages [12233.190901] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12233.207916] Lustre: Skipped 3 previous similar messages [12238.309284] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12238.319839] Lustre: Skipped 3 previous similar messages [12240.351221] Lustre: tst250fs-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12240.754117] Lustre: server umount tst250fs-MDT0000 complete [12243.868993] LustreError: 373671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854721 with bad export cookie 12934625812679993590 [12243.872314] LustreError: MGC192.168.206.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12243.881784] LustreError: 373671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12244.234371] Lustre: server umount tst250fs-MDT0001 complete [12257.863792] Lustre: server umount tst250fs-OST0000 complete [12272.058988] Lustre: server umount tst250fs-OST0001 complete [12290.870380] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_hostid [12298.112622] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [12327.120813] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 14:20:02 (1787854802) [12328.626572] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [12344.505947] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing set_hostid [12354.275939] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing load_modules_local [12381.423578] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12106 sec ========= 14:20:57 (1787854857) [12382.685539] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 14:20:58 (1787854858) === [12385.556723] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 14:21:01 (1787854861) === [12419.545522] Lustre: DEBUG MARKER: oleg626-server.virtnet: executing unload_modules_local [12421.980734] Key type lgssc unregistered [12422.269172] LNet: 385326:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12422.279952] LNetError: 385326:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12423.339425] LNet: Removed LNI 192.168.206.126@tcp [12424.194145] Key type .llcrypt unregistered [12424.195780] Key type ._llcrypt unregistered