[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 554478994 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a2000-0x1465ccfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001016] APIC: Switch to symmetric I/O mode setup [ 0.002333] x2apic enabled [ 0.003011] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.007508] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.008030] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.009016] pid_max: default: 32768 minimum: 301 [ 0.010160] LSM: Security Framework initializing [ 0.011057] Yama: becoming mindful. [ 0.012071] SELinux: Initializing. [ 0.013067] *** VALIDATE selinux *** [ 0.022065] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027096] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028158] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030075] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031128] *** VALIDATE tmpfs *** [ 0.033228] *** VALIDATE proc *** [ 0.034295] *** VALIDATE cgroup *** [ 0.035010] *** VALIDATE cgroup2 *** [ 0.037080] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038169] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040033] Spectre V2 : User space: Vulnerable [ 0.041013] Speculative Store Bypass: Vulnerable [ 0.044712] debug: unmapping init [mem 0xffffffff89459000-0xffffffff89460fff] [ 0.047255] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048844] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049027] ... version: 2 [ 0.050015] ... bit width: 48 [ 0.051011] ... generic registers: 4 [ 0.052014] ... value mask: 0000ffffffffffff [ 0.053016] ... max period: 00007fffffffffff [ 0.054019] ... fixed-purpose events: 3 [ 0.055011] ... event mask: 000000070000000f [ 0.057372] rcu: Hierarchical SRCU implementation. [ 0.059890] smp: Bringing up secondary CPUs ... [ 0.060606] x86: Booting SMP configuration: [ 0.061134] .... node #0, CPUs: #1 #2 #3 [ 0.071143] smp: Brought up 1 node, 4 CPUs [ 0.073021] smpboot: Max logical packages: 1 [ 0.074015] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.109673] node 0 deferred pages initialised in 33ms [ 0.114200] devtmpfs: initialized [ 0.115362] x86/mm: Memory block size: 128MB [ 0.118260] gcov: version magic: 0x41383552 [ 0.120542] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.121191] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.122557] pinctrl core: initialized pinctrl subsystem [ 0.123236] [ 0.123709] ************************************************************* [ 0.124017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.125020] ** ** [ 0.126014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.127014] ** ** [ 0.128017] ** This means that this kernel is built to expose internal ** [ 0.129014] ** IOMMU data structures, which may compromise security on ** [ 0.130015] ** your system. ** [ 0.131014] ** ** [ 0.132016] ** If you see this message and you are not debugging the ** [ 0.133013] ** kernel, report this immediately to your vendor! ** [ 0.134073] ** ** [ 0.135015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.136016] ************************************************************* [ 0.137763] NET: Registered protocol family 16 [ 0.139713] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.142077] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.145079] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.148869] cpuidle: using governor menu [ 0.151165] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.153756] PCI: Using configuration type 1 for base access [ 0.156151] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.163218] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.164017] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.167115] cryptd: max_cpu_qlen set to 1000 [ 0.169283] ACPI: Added _OSI(Module Device) [ 0.170020] ACPI: Added _OSI(Processor Device) [ 0.171015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.172027] ACPI: Added _OSI(Processor Aggregator Device) [ 0.176885] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.186459] ACPI: Interpreter enabled [ 0.188080] ACPI: PM: (supports S0 S3 S4 S5) [ 0.189014] ACPI: Using IOAPIC for interrupt routing [ 0.191163] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.196544] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.211148] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.213048] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.217069] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.220100] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.227768] acpiphp: Slot [2] registered [ 0.229260] acpiphp: Slot [5] registered [ 0.231283] acpiphp: Slot [6] registered [ 0.233324] acpiphp: Slot [7] registered [ 0.235136] acpiphp: Slot [8] registered [ 0.237186] acpiphp: Slot [9] registered [ 0.238168] acpiphp: Slot [10] registered [ 0.240216] acpiphp: Slot [3] registered [ 0.242181] acpiphp: Slot [4] registered [ 0.244244] acpiphp: Slot [11] registered [ 0.246173] acpiphp: Slot [12] registered [ 0.247228] acpiphp: Slot [13] registered [ 0.249232] acpiphp: Slot [14] registered [ 0.251126] acpiphp: Slot [15] registered [ 0.253150] acpiphp: Slot [16] registered [ 0.255168] acpiphp: Slot [17] registered [ 0.257196] acpiphp: Slot [18] registered [ 0.259159] acpiphp: Slot [19] registered [ 0.261221] acpiphp: Slot [20] registered [ 0.263315] acpiphp: Slot [21] registered [ 0.264338] acpiphp: Slot [22] registered [ 0.266324] acpiphp: Slot [23] registered [ 0.268138] acpiphp: Slot [24] registered [ 0.270980] acpiphp: Slot [25] registered [ 0.273253] acpiphp: Slot [26] registered [ 0.275226] acpiphp: Slot [27] registered [ 0.277250] acpiphp: Slot [28] registered [ 0.279189] acpiphp: Slot [29] registered [ 0.281340] acpiphp: Slot [30] registered [ 0.284136] acpiphp: Slot [31] registered [ 0.286101] PCI host bridge to bus 0000:00 [ 0.289040] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.292061] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.295037] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.298098] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.301035] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.305039] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.308334] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.311506] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.315780] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.329023] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.333756] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.336064] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.338000] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.340084] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.343693] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.345968] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.349052] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.354096] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.359017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.372019] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.377023] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.384893] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.407023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.424021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.447021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.460569] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.470021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.481020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.498023] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.517063] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.524021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.530032] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.541000] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.553000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.562019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.570015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.594020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.606016] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.614024] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.623027] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.643019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.650837] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.658019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.664019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.680024] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.689693] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.691548] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.695511] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.698490] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.701344] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.707010] iommu: Default domain type: Passthrough [ 0.708378] SCSI subsystem initialized [ 0.709158] ACPI: bus type USB registered [ 0.711117] usbcore: registered new interface driver usbfs [ 0.713086] usbcore: registered new interface driver hub [ 0.714102] usbcore: registered new device driver usb [ 0.716334] pps_core: LinuxPPS API ver. 1 registered [ 0.718014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.722089] PTP clock support registered [ 0.724250] EDAC MC: Ver: 3.0.0 [ 0.725115] PCI: Using ACPI for IRQ routing [ 0.726000] NetLabel: Initializing [ 0.728013] NetLabel: domain hash size = 128 [ 0.729008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.730281] NetLabel: unlabeled traffic allowed by default [ 0.733463] vgaarb: loaded [ 0.735780] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.737013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.744763] clocksource: Switched to clocksource kvm-clock [ 0.902881] VFS: Disk quotas dquot_6.6.0 [ 0.905576] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.908977] *** VALIDATE ramfs *** [ 0.910856] *** VALIDATE hugetlbfs *** [ 0.913577] pnp: PnP ACPI init [ 0.918768] pnp: PnP ACPI: found 6 devices [ 0.946732] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.950787] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.953349] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.956125] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.960081] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.963016] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.966215] NET: Registered protocol family 2 [ 0.968987] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.974860] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.978857] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.984569] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.987868] TCP: Hash tables configured (established 65536 bind 65536) [ 0.990910] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.993961] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.997392] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.000766] NET: Registered protocol family 1 [ 1.004222] RPC: Registered named UNIX socket transport module. [ 1.006818] RPC: Registered udp transport module. [ 1.008768] RPC: Registered tcp transport module. [ 1.010621] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.013344] NET: Registered protocol family 44 [ 1.014969] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.018244] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.021156] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.024867] PCI: CLS 0 bytes, default 64 [ 1.027483] Unpacking initramfs... [ 3.078865] debug: unmapping init [mem 0xffff967f7cc54000-0xffff967f7ffbffff] [ 3.083685] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.086173] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.089437] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 4.617516] Initialise system trusted keyrings [ 4.619893] Key type blacklist registered [ 4.621257] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.629138] zbud: loaded [ 4.631587] *** VALIDATE nfs *** [ 4.632462] *** VALIDATE nfs4 *** [ 4.633435] pstore: using deflate compression [ 4.636222] Platform Keyring initialized [ 4.723949] NET: Registered protocol family 38 [ 4.725404] Key type asymmetric registered [ 4.726487] Asymmetric key parser 'x509' registered [ 4.729200] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.731489] io scheduler mq-deadline registered [ 4.732648] io scheduler kyber registered [ 4.733937] io scheduler bfq registered [ 4.735324] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.737560] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.739687] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.742256] ACPI: Power Button [PWRF] [ 4.748299] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.754667] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.767709] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.776697] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.812318] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.842500] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.875673] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.880760] Non-volatile memory driver v1.3 [ 4.882595] Linux agpgart interface v0.103 [ 4.911690] virtio_blk virtio1: [vda] 149936 512-byte logical blocks (76.8 MB/73.2 MiB) [ 4.914965] vda: detected capacity change from 0 to 76767232 [ 4.930672] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.933722] vdb: detected capacity change from 0 to 1073741824 [ 4.960944] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.968668] vdc: detected capacity change from 0 to 2621440000 [ 4.995330] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.998658] vdd: detected capacity change from 0 to 2621440000 [ 5.112868] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.115730] vde: detected capacity change from 0 to 4294967296 [ 5.132107] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.135800] vdf: detected capacity change from 0 to 4294967296 [ 5.146704] libphy: Fixed MDIO Bus: probed [ 5.157876] usbcore: registered new interface driver usbserial_generic [ 5.162574] usbserial: USB Serial support registered for generic [ 5.167397] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.175554] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.178877] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.181933] mousedev: PS/2 mouse device common for all mice [ 5.184957] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.188655] rtc_cmos 00:05: RTC can wake from S4 [ 5.194401] rtc_cmos 00:05: registered as rtc0 [ 5.198777] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.203814] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.261278] hpet1: lost 3 rtc interrupts [ 5.272650] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.287935] intel_pstate: CPU model not supported [ 5.295717] hid: raw HID events driver (C) Jiri Kosina [ 5.297560] usbcore: registered new interface driver usbhid [ 5.299335] usbhid: USB HID core driver [ 5.300824] drop_monitor: Initializing network drop monitor service [ 5.303203] Initializing XFRM netlink socket [ 5.304807] NET: Registered protocol family 10 [ 5.307365] Segment Routing with IPv6 [ 5.308556] NET: Registered protocol family 17 [ 5.310372] mpls_gso: MPLS GSO support [ 5.316228] RAS: Correctable Errors collector initialized. [ 5.317910] AVX version of gcm_enc/dec engaged. [ 5.319358] AES CTR mode by8 optimization enabled [ 5.408749] sched_clock: Marking stable (5408677089, 0)->(6765226596, -1356549507) [ 5.415582] registered taskstats version 1 [ 5.417594] Loading compiled-in X.509 certificates [ 5.419934] zswap: loaded using pool lzo/zbud [ 5.470856] Key type big_key registered [ 5.499732] Key type encrypted registered [ 5.501156] ima: No TPM chip found, activating TPM-bypass! [ 5.509498] ima: Allocated hash algorithm: sha1 [ 5.514073] ima: No architecture policies found [ 5.517551] evm: Initialising EVM extended attributes: [ 5.520686] evm: security.selinux [ 5.522580] evm: security.ima [ 5.527224] evm: security.capability [ 5.528823] evm: HMAC attrs: 0x1 [ 5.532575] rtc_cmos 00:05: setting system clock to 2026-09-08 12:09:06 UTC (1788869346) [ 5.542904] debug: unmapping init [mem 0xffffffff8a403000-0xffffffff8a5fffff] [ 5.548140] debug: unmapping init [mem 0xffffffff89182000-0xffffffff89458fff] [ 5.561230] Write protecting the kernel read-only data: 28672k [ 5.564079] debug: unmapping init [mem 0xffffffff87803000-0xffffffff879fffff] [ 5.569681] debug: unmapping init [mem 0xffffffff88114000-0xffffffff881fffff] [ 5.651889] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.670619] systemd[1]: Detected virtualization kvm. [ 5.675442] systemd[1]: Detected architecture x86-64. [ 5.680567] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.716311] systemd[1]: No hostname configured. [ 5.720275] systemd[1]: Set hostname to . [ 5.724688] random: systemd: uninitialized urandom read (16 bytes read) [ 5.728256] systemd[1]: Initializing machine ID from random generator. [ 5.874399] random: ln: uninitialized urandom read (6 bytes read) [ 6.066810] random: systemd: uninitialized urandom read (16 bytes read) [ 6.070663] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 6.076761] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 6.082905] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.075426] device-mapper: uevent: version 1.0.3 [ 7.077459] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 8.945752] virtio_net virtio0 ens2: renamed from eth0 [ 9.009310] random: fast init done [ 10.036413] scsi host0: ata_piix [ 10.233548] scsi host1: ata_piix [ 10.240846] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 10.258098] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 15.039692] random: crng init done [ 15.041422] random: 7 urandom warning(s) missed due to ratelimiting [ 17.865327] dracut-initqueue[595]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 19.306699] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 21.156501] printk: systemd: 26 output lines suppressed due to ratelimiting [ 21.795040] SELinux: Disabled at runtime. [ 21.928277] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 21.948604] systemd[1]: Detected virtualization kvm. [ 21.950493] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 23.262949] systemd[1]: initrd-switch-root.service: Succeeded. [ 23.271638] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 23.306536] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 23.324205] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 23.332313] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 23.357722] systemd[1]: Starting Journal Service... Starting Journal Service... [ 23.363331] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. [ OK ] Stopped target Switch Root. [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target Slices. Mounting Kernel Debug File System... [ 23.465859] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target RPC Port Mapper. Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... Starting Apply Kernel Variables... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Local Encrypted Volumes. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 24.531996] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 25.581323] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 25.719449] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 26.253698] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 26.340416] EDAC sbridge: Ver: 1.1.2 [ 29.059462] Key type dns_resolver registered [ 29.920933] NFS: Registering the id_resolver key type [ 29.925545] Key type id_resolver registered [ 29.928425] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (6s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg150-server login: [ 70.440619] libcfs: loading out-of-tree module taints kernel. [ 70.461440] Key type ._llcrypt registered [ 70.462819] Key type .llcrypt registered [ 70.521857] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_hostid [ 80.220315] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 80.884513] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 80.893563] alg: No test for adler32 (adler32-zlib) [ 81.957423] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 82.361861] LNet: Added LNI 192.168.201.150@tcp [8/256/0/180] [ 84.031363] Key type lgssc registered [ 84.760952] Lustre: Echo OBD driver; http://www.lustre.org/ [ 93.435854] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 114.219594] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 121.463788] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 121.475926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 122.652847] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 122.680722] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 122.748212] Lustre: lustre-MDT0000: new disk, initializing [ 122.786296] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 122.793882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 125.672823] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 134.300440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 134.351680] Lustre: 6488:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 134.374316] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 134.376943] Lustre: Skipped 1 previous similar message [ 134.431606] Lustre: lustre-MDT0001: new disk, initializing [ 134.469391] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 134.483286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 134.492404] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 136.776320] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 139.732335] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 144.455103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 144.577716] Lustre: lustre-OST0000: new disk, initializing [ 144.580823] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 144.585146] Lustre: 8426:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 144.610913] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 146.484837] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 146.488129] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 146.517596] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 147.763257] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 160.447695] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 160.704523] Lustre: lustre-OST0001: new disk, initializing [ 160.719619] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 160.733830] Lustre: 9498:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 160.887207] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 169.133481] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 170.053224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 170.076977] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 170.195276] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 182.808509] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 191.915140] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 200.097608] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing check_logdir /tmp/testlogs/ [ 205.575310] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing yml_node [ 209.711508] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 211.859683] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 213.877204] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 215.304838] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 8 08:12:34 EDT 2026 [ 230.498613] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 232.338311] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 246.028110] hrtimer: interrupt took 6099084 ns [ 252.385140] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 252.385899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 252.386799] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 252.413826] Lustre: Skipped 3 previous similar messages [ 252.448688] Lustre: Skipped 3 previous similar messages [ 256.365852] Lustre: server umount lustre-MDT0000 complete [ 259.243497] LustreError: 6481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788869600 with bad export cookie 12172153372343954365 [ 259.250810] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 259.256327] LustreError: 6481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 262.624023] LustreError: 8433:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 262.627913] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 262.635489] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 262.638789] LustreError: 8433:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 262.653512] Lustre: Skipped 1 previous similar message [ 264.673088] LustreError: 6496:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 264.677838] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 264.693282] LustreError: 6496:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 264.708495] Lustre: Skipped 2 previous similar messages [ 265.647476] Lustre: server umount lustre-MDT0001 complete [ 269.263973] Lustre: server umount lustre-OST0000 complete [ 273.230250] Lustre: server umount lustre-OST0001 complete [ 282.392389] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_hostid [ 298.158769] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 336.010724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 336.286963] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 336.326637] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 336.439737] Lustre: lustre-MDT0000: new disk, initializing [ 336.552150] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 336.582117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 341.161559] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 352.132760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 352.232712] Lustre: 15067:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 352.274970] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 352.279107] Lustre: Skipped 1 previous similar message [ 352.340530] Lustre: lustre-MDT0001: new disk, initializing [ 352.416404] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 352.481287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 352.490312] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 356.897121] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 365.589622] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 371.019101] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 377.492620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 377.871149] Lustre: lustre-OST0000: new disk, initializing [ 377.874628] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 377.882522] Lustre: 17041:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 377.960626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 379.412783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 379.433881] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 379.550878] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 384.062281] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 393.780787] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 398.260120] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 398.448570] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 402.525255] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 402.741949] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 405.473529] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 405.474400] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 405.480459] LustreError: Skipped 1 previous similar message [ 405.481693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 405.492756] Lustre: Skipped 1 previous similar message [ 410.603515] Lustre: server umount lustre-OST0000 complete [ 414.176644] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 414.180938] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 414.189418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 414.192983] Lustre: Skipped 1 previous similar message [ 420.487684] Lustre: server umount lustre-MDT0000 complete [ 420.834088] LustreError: 15882:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 420.860210] LustreError: 15882:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 424.197934] LustreError: 15061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788869765 with bad export cookie 12172153372343955898 [ 424.215670] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 424.221541] LustreError: 15061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 424.536538] Lustre: server umount lustre-MDT0001 complete [ 430.032125] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 08:16:09 (1788869769) [ 447.300024] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 447.775073] LustreError: 19916:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 447.846134] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 451.984918] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 453.100228] LustreError: 19917:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 460.592496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 465.366587] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 473.628154] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 479.717993] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 481.780089] Lustre: Failing over lustre-MDT0000 [ 482.011938] Lustre: server umount lustre-MDT0000 complete [ 482.788280] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 482.799163] Lustre: Skipped 1 previous similar message [ 482.800612] LustreError: 19912:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 482.836985] LustreError: 19912:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 498.149579] Lustre: 3628:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788869823/real 1788869823] req@ffff967fe6ed8e00 x1875765544244480/t0(0) o400->MGC192.168.201.150@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788869839 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 498.169619] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 501.411564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 503.264274] LustreError: 19916:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 503.282880] LustreError: 19916:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 508.384151] LustreError: 3627:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff967fe6ed9880 x1875765544247424/t0(0) o250->MGC192.168.201.150@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 514.031157] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 514.035864] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 514.055792] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 514.082344] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 524.644110] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 529.377809] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 529.385423] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 529.398091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 529.406684] Lustre: Skipped 2 previous similar messages [ 534.879866] Lustre: server umount lustre-MDT0000 complete [ 536.544893] LustreError: 19916:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 536.573767] LustreError: 19916:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 538.659497] LustreError: 19895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788869879 with bad export cookie 12172153372343957312 [ 538.677948] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 538.936733] Lustre: server umount lustre-MDT0001 complete [ 547.486561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 547.897572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 547.904416] Lustre: Skipped 2 previous similar messages [ 552.096814] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 561.509386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 566.503739] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 574.671983] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 581.724643] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 584.344818] Lustre: Failing over lustre-MDT0000 [ 584.715945] Lustre: server umount lustre-MDT0000 complete [ 586.419220] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 586.439458] Lustre: Skipped 1 previous similar message [ 602.080664] Lustre: 3631:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788869927/real 1788869927] req@ffff967ec31e5c00 x1875765544275328/t0(0) o400->MGC192.168.201.150@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788869943 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 602.082259] LustreError: 23981:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 602.105243] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 602.131399] LustreError: 23981:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 604.370693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 613.826440] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 618.040411] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 618.987970] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 619.005410] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 619.015829] Lustre: Skipped 1 previous similar message [ 619.036555] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 628.806391] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 634.338172] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 634.350334] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 634.365041] Lustre: Skipped 1 previous similar message [ 634.374886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 634.377569] Lustre: Skipped 2 previous similar messages [ 639.623543] Lustre: server umount lustre-MDT0000 complete [ 643.954802] LustreError: 23234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788869984 with bad export cookie 12172153372343958250 [ 643.966939] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 643.967662] LustreError: 23234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 644.254789] Lustre: server umount lustre-MDT0001 complete [ 660.793253] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 08:19:58 (1788869998) [ 685.546243] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_hostid [ 697.430484] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 749.248750] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 760.598058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 760.949820] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 760.980988] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 761.098279] Lustre: lustre-MDT0000: new disk, initializing [ 761.211580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 761.215780] Lustre: Skipped 2 previous similar messages [ 761.234564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 766.027355] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 777.511923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 777.687611] Lustre: 29630:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 777.884214] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 782.392805] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 792.317865] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 800.048595] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 808.877685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 809.192880] Lustre: 31597:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 810.354034] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 810.478611] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 810.568408] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 810.590921] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 815.908452] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 827.233811] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 834.137169] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 856.548160] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 856.550132] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 856.566717] Lustre: Skipped 1 previous similar message [ 856.576105] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 856.593540] Lustre: Skipped 3 previous similar messages [ 861.671766] LustreError: 31583:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 861.681827] LustreError: 31583:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 861.858551] Lustre: server umount lustre-OST0000 complete [ 872.026455] Lustre: server umount lustre-MDT0000 complete [ 876.176319] LustreError: 29622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788870217 with bad export cookie 12172153372343958656 [ 876.181142] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 876.185796] LustreError: 29622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 876.611635] Lustre: server umount lustre-MDT0001 complete [ 888.674561] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [ 891.758403] Key type lgssc unregistered [ 892.102145] LNet: 33576:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 892.114387] LNetError: 33576:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 892.132626] LNet: Removed LNI 192.168.201.150@tcp [ 893.391244] Key type .llcrypt unregistered [ 893.393474] Key type ._llcrypt unregistered [ 923.144577] Key type ._llcrypt registered [ 923.146871] Key type .llcrypt registered [ 923.279789] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 924.694073] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 924.913518] alg: No test for adler32 (adler32-zlib) [ 926.069121] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 926.307885] LNet: Added LNI 192.168.201.150@tcp [8/256/0/180] [ 927.959416] Key type lgssc registered [ 929.103268] Lustre: Echo OBD driver; http://www.lustre.org/ [ 936.805313] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 936.872621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 938.490163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 944.137183] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 948.706142] LustreError: 34992:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 948.726734] LustreError: 34992:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 953.339965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 953.639056] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 959.129037] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 967.676837] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 974.614842] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 982.487373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 983.090219] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 990.420270] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 998.541035] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 999.561660] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1015.249413] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 08:25:54 (1788870354) [ 1017.188496] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1018.992859] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1027.421922] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 08:26:06 (1788870366) [ 1037.170802] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 08:26:16 (1788870376) [ 1038.166622] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1047.521345] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1047.526430] Lustre: Skipped 2 previous similar messages [ 1049.883538] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 08:26:28 (1788870388) [ 1052.255825] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1052.262022] Lustre: Skipped 2 previous similar messages [ 1060.678422] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1060.686476] Lustre: Skipped 4 previous similar messages [ 1076.572953] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 08:26:55 (1788870415) [ 1079.765319] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1083.832995] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1092.985733] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 08:27:12 (1788870432) [ 1095.079539] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1095.082662] Lustre: Skipped 6 previous similar messages [ 1109.838527] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 08:27:28 (1788870448) [ 1126.882424] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1126.887371] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1126.894573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1126.914020] Lustre: Skipped 1 previous similar message [ 1131.196257] Lustre: server umount lustre-OST0000 complete [ 1137.138764] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1137.165549] Lustre: Skipped 1 previous similar message [ 1137.173840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1142.243929] LustreError: 34992:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1142.258745] LustreError: 34992:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1143.850439] Lustre: server umount lustre-MDT0000 complete [ 1146.346462] LustreError: 34986:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1146.373278] LustreError: 34986:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1147.520932] LustreError: 38555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788870488 with bad export cookie 9017943752220053301 [ 1147.528604] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1147.532854] LustreError: 38555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1148.005596] Lustre: server umount lustre-MDT0001 complete [ 1157.931222] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [ 1160.274416] Key type lgssc unregistered [ 1160.610912] LNet: 42844:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1160.621862] LNetError: 42844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1160.653834] LNet: Removed LNI 192.168.201.150@tcp [ 1161.678712] Key type .llcrypt unregistered [ 1161.685584] Key type ._llcrypt unregistered [ 1180.279857] Key type ._llcrypt registered [ 1180.282576] Key type .llcrypt registered [ 1180.430660] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_hostid [ 1198.925281] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 1200.489532] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1200.511851] alg: No test for adler32 (adler32-zlib) [ 1201.555053] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 1201.892399] LNet: Added LNI 192.168.201.150@tcp [8/256/0/180] [ 1203.591291] Key type lgssc registered [ 1204.981445] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1242.104571] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1242.147140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1243.475718] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1243.500124] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1243.591265] Lustre: lustre-MDT0000: new disk, initializing [ 1243.695861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1243.730215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1248.139342] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 1258.990186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1259.180512] Lustre: 45992:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1259.255294] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1259.259622] Lustre: Skipped 1 previous similar message [ 1259.358957] Lustre: lustre-MDT0001: new disk, initializing [ 1259.485330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1259.543861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1259.559793] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1264.583482] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 1273.554457] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1279.110089] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1286.981885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1287.252204] Lustre: lustre-OST0000: new disk, initializing [ 1287.254677] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1287.261728] Lustre: 47961:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1287.315374] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1293.021846] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 1296.963522] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1296.973965] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1297.071620] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1302.869918] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1307.956305] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1308.182419] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1313.763374] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1313.983395] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1317.861928] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1317.868300] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1317.876453] LustreError: Skipped 1 previous similar message [ 1317.879389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1317.923316] Lustre: Skipped 1 previous similar message [ 1322.980551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1322.987594] Lustre: Skipped 1 previous similar message [ 1328.105118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1330.143429] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1330.443244] Lustre: server umount lustre-OST0000 complete [ 1336.293357] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1336.303453] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1336.318779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1336.331856] Lustre: Skipped 1 previous similar message [ 1338.341658] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1340.729588] Lustre: server umount lustre-MDT0000 complete [ 1344.904778] LustreError: 45983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788870685 with bad export cookie 10657519897716513221 [ 1344.923297] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1345.278755] Lustre: server umount lustre-MDT0001 complete [ 1355.593192] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 08:31:34 (1788870694) [ 1363.578390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1363.975785] LustreError: 50424:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1363.990640] LustreError: 50424:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1364.041536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1368.375756] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 1369.060825] LustreError: 50424:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1369.076784] LustreError: 50424:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1373.179842] LustreError: 50425:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1376.632747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1376.927692] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1381.592627] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 1388.990329] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1394.783719] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1401.331229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1401.612828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1407.513130] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 1415.358981] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1420.083499] LustreError: 52953:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1421.240919] Lustre: Modifying parameter general.timeout=1 in log params [ 1423.121091] Lustre: Modifying parameter general.timeout=3 in log params [ 1423.123935] Lustre: Skipped 1 previous similar message [ 1425.473259] Lustre: Modifying parameter general.timeout=6 in log params [ 1425.475889] Lustre: Skipped 2 previous similar messages [ 1429.684515] Lustre: Modifying parameter general.timeout=11 in log params [ 1429.689300] Lustre: Skipped 4 previous similar messages [ 1438.303263] Lustre: Modifying parameter general.timeout=21 in log params [ 1438.307569] Lustre: Skipped 9 previous similar messages [ 1454.506238] Lustre: Modifying parameter general.timeout=40 in log params [ 1454.511766] Lustre: Skipped 18 previous similar messages [ 1486.545857] Lustre: Modifying parameter general.timeout=77 in log params [ 1486.550609] Lustre: Skipped 36 previous similar messages [ 1542.077241] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 08:34:41 (1788870881) [ 1545.041414] Lustre: 59888:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1547.180471] Lustre: 59999:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1550.931198] Lustre: 60196:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1550.956376] Lustre: 60196:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1551.852951] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=50 in log params [ 1551.859901] Lustre: Skipped 58 previous similar messages [ 1577.340804] Lustre: 61749:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1577.345550] Lustre: 61749:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1577.360496] LustreError: 61749:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1580.847779] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 08:35:20 (1788870920) [ 1591.777714] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1591.788353] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1591.809464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1591.822599] Lustre: Skipped 2 previous similar messages [ 1597.758921] Lustre: server umount lustre-MDT0000 complete [ 1599.972505] LustreError: 50420:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1599.988344] LustreError: 50420:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1601.294145] LustreError: 50406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788870942 with bad export cookie 10657519897716514103 [ 1601.295947] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1601.301837] LustreError: 50406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 1601.565711] Lustre: server umount lustre-MDT0001 complete [ 1615.184193] Lustre: server umount lustre-OST0000 complete [ 1641.836904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1642.032640] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1642.233975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1646.524756] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 1654.285161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1654.356983] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1654.401883] Lustre: 64283:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1659.393943] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 1666.901750] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1672.561697] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1679.182765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1679.387053] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1679.661995] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1679.676400] Lustre: Skipped 1 previous similar message [ 1684.820516] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 1691.648771] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1693.159304] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1693.168554] Lustre: Skipped 17 previous similar messages [ 1693.219388] Lustre: 66848:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1702.371348] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1702.377836] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1702.393845] Lustre: Skipped 2 previous similar messages [ 1702.398449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1702.401134] Lustre: Skipped 3 previous similar messages [ 1705.704665] Lustre: server umount lustre-OST0000 complete [ 1711.072557] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1711.084762] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1711.084898] LustreError: Skipped 1 previous similar message [ 1711.104936] Lustre: Skipped 1 previous similar message [ 1715.484475] Lustre: server umount lustre-MDT0000 complete [ 1718.256095] LustreError: 65123:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1718.275330] LustreError: 65123:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1718.922786] LustreError: 64273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788871059 with bad export cookie 10657519897716517022 [ 1718.929973] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1718.930976] LustreError: 64273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1719.216864] Lustre: server umount lustre-MDT0001 complete [ 1729.495639] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [ 1731.952880] Key type lgssc unregistered [ 1732.272614] LNet: 68054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1732.288315] LNetError: 68054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1732.311964] LNet: Removed LNI 192.168.201.150@tcp [ 1733.405858] Key type .llcrypt unregistered [ 1733.408828] Key type ._llcrypt unregistered [ 1759.554527] Key type ._llcrypt registered [ 1759.556881] Key type .llcrypt registered [ 1759.637555] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 1760.565787] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1760.677806] alg: No test for adler32 (adler32-zlib) [ 1761.825433] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 1762.131857] LNet: Added LNI 192.168.201.150@tcp [8/256/0/180] [ 1763.872743] Key type lgssc registered [ 1765.292731] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1773.707459] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1773.735285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1775.320437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1780.247981] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 1785.829409] LustreError: 69465:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1785.866819] LustreError: 69465:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1789.697106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1790.413682] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1794.601161] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 1801.598345] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1809.014408] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1815.921969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1816.054507] LustreError: 71293:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1816.625651] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1823.152923] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 1829.880688] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1836.929154] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1846.241137] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1846.250819] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1846.253706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1851.361918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1851.369616] Lustre: Skipped 4 previous similar messages [ 1856.483560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1856.487553] Lustre: Skipped 2 previous similar messages [ 1859.039546] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1859.224203] Lustre: server umount lustre-MDT0000 complete [ 1861.604171] LustreError: 69464:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1861.624890] LustreError: 69464:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1863.072670] LustreError: 69444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788871204 with bad export cookie 12540350660581213863 [ 1863.078759] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1863.087290] LustreError: 69444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1863.399646] Lustre: server umount lustre-MDT0001 complete [ 1877.810706] Lustre: server umount lustre-OST0000 complete [ 1903.052860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1903.425963] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1903.454501] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1903.702983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1908.081499] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 1917.501923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1917.642510] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1917.685602] Lustre: 74337:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1918.124674] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1923.026036] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 1930.681246] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1936.626350] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1943.528289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1943.737369] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1944.140784] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1950.478777] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 1958.031607] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1959.841305] Lustre: 76912:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1961.710214] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1967.076598] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1967.087668] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1967.097492] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1967.110981] Lustre: Skipped 3 previous similar messages [ 1967.125907] Lustre: Skipped 1 previous similar message [ 1972.195513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1972.199870] Lustre: Skipped 1 previous similar message [ 1973.314418] Lustre: server umount lustre-OST0000 complete [ 1979.360338] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1979.366861] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1982.434445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1982.437066] Lustre: Skipped 1 previous similar message [ 1982.438091] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1983.700808] Lustre: server umount lustre-MDT0000 complete [ 1986.666140] LustreError: 76915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788871327 with bad export cookie 12540350660581214710 [ 1986.668654] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1986.674427] LustreError: 76915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1986.895315] Lustre: server umount lustre-MDT0001 complete [ 1996.778295] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [ 1998.944378] Key type lgssc unregistered [ 1999.235683] LNet: 78125:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1999.239734] LNetError: 78125:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2000.298232] LNet: Removed LNI 192.168.201.150@tcp [ 2001.156193] Key type .llcrypt unregistered [ 2001.159715] Key type ._llcrypt unregistered [ 2026.263167] Key type ._llcrypt registered [ 2026.265432] Key type .llcrypt registered [ 2026.363899] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 2027.581393] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2027.694419] alg: No test for adler32 (adler32-zlib) [ 2028.787620] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 2029.174780] LNet: Added LNI 192.168.201.150@tcp [8/256/0/180] [ 2030.935312] Key type lgssc registered [ 2032.163297] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2039.890479] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2039.925731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2041.711312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2046.869274] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 2051.055212] LustreError: 79536:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2051.085377] LustreError: 79536:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2055.457341] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2055.963699] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2060.748823] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 2067.896943] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2073.505693] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2079.810424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2080.290427] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2086.632266] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 2094.510282] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2101.144173] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2102.206266] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2103.206400] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2106.102162] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2106.114283] Lustre: Skipped 2 previous similar messages [ 2110.914228] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2110.916951] Lustre: Skipped 4 previous similar messages [ 2119.460429] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2119.470786] Lustre: Skipped 7 previous similar messages [ 2136.096766] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2136.104986] Lustre: Skipped 16 previous similar messages [ 2168.980547] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2168.987071] Lustre: Skipped 33 previous similar messages [ 2206.872727] Lustre: 87640:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2209.282296] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 08:45:48 (1788871548) [ 2211.248861] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2213.433547] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 08:45:52 (1788871552) [ 2219.497438] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2219.498340] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2219.500665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2219.544441] Lustre: Skipped 1 previous similar message [ 2224.611689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2224.624809] Lustre: Skipped 2 previous similar messages [ 2229.729246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2233.311146] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2233.642717] Lustre: server umount lustre-OST0000 complete [ 2239.968788] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2239.975082] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2239.981106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2239.981114] Lustre: Skipped 1 previous similar message [ 2240.013062] Lustre: Skipped 1 previous similar message [ 2244.376592] Lustre: server umount lustre-MDT0000 complete [ 2245.096565] LustreError: 79530:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2245.119391] LustreError: 79530:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2248.133673] LustreError: 81384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788871589 with bad export cookie 3317561261708691419 [ 2248.135795] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2248.149058] LustreError: 81384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2248.585175] Lustre: server umount lustre-MDT0001 complete [ 2259.842266] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [ 2262.451502] Key type lgssc unregistered [ 2262.677264] LNet: 88889:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2262.685255] LNetError: 88889:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2262.694612] LNet: Removed LNI 192.168.201.150@tcp [ 2263.595408] Key type .llcrypt unregistered [ 2263.597702] Key type ._llcrypt unregistered [ 2267.996627] Key type ._llcrypt registered [ 2268.002540] Key type .llcrypt registered [ 2268.158825] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2272.332228] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules [ 2273.263810] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2273.344035] alg: No test for adler32 (adler32-zlib) [ 2274.314563] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 2274.543158] LNet: Added LNI 192.168.201.150@tcp [8/256/0/180] [ 2276.151244] Key type lgssc registered [ 2277.132441] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2278.003456] LustreError: 90087:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2295.370236] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 2300.105735] LustreError: 90087:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2305.860638] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2305.879466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2307.500676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2311.922672] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 2317.796274] LustreError: 91128:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2317.812722] LustreError: 91128:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2321.707739] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 08:47:40 (1788871660) [ 2322.915669] LustreError: 91129:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2327.267497] LustreError: 91160:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2327.281891] LustreError: 91160:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2327.292441] LustreError: 91160:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 2327.741970] Lustre: server umount lustre-MDT0000 complete [ 2343.159542] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [ 2346.170501] Key type lgssc unregistered [ 2346.431445] LNet: 92653:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2346.439133] LNetError: 92653:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2346.474861] LNet: Removed LNI 192.168.201.150@tcp [ 2347.479180] Key type .llcrypt unregistered [ 2347.480875] Key type ._llcrypt unregistered [ 2367.689558] Key type ._llcrypt registered [ 2367.694764] Key type .llcrypt registered [ 2367.886588] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 2369.543224] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2369.577606] alg: No test for adler32 (adler32-zlib) [ 2370.672977] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 2370.930424] LNet: Added LNI 192.168.201.150@tcp [8/256/0/180] [ 2372.631656] Key type lgssc registered [ 2373.997620] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2382.132102] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2382.154407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2383.708043] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2388.246992] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 2393.065926] LustreError: 93880:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2393.096348] LustreError: 93880:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2397.515180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2398.159090] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2402.759856] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 2409.787568] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2415.529820] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2422.933369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2423.795386] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2430.465988] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 2438.724199] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2489.839622] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 08:50:29 (1788871829) [ 2491.547294] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2493.461836] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 08:50:32 (1788871832) [ 2498.530336] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2498.536386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2498.561375] Lustre: Skipped 1 previous similar message [ 2498.571962] Lustre: Skipped 2 previous similar messages [ 2503.177983] Lustre: server umount lustre-MDT0000 complete [ 2503.648500] LustreError: 93880:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2503.665429] LustreError: 93880:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2506.612881] LustreError: 93860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788871847 with bad export cookie 8098744978293873369 [ 2506.636436] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2506.636777] LustreError: 93860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2506.943376] Lustre: server umount lustre-MDT0001 complete [ 2520.947828] Lustre: server umount lustre-OST0000 complete [ 2532.741435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2533.083929] LustreError: 98172:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2533.210338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2537.292171] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 2538.465589] LustreError: 98173:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2543.586587] LustreError: 98172:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2545.879246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2546.764186] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2552.359854] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 2560.448932] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2566.307747] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2579.942404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2580.115706] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2580.121444] LustreError: 98162:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2580.161339] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2580.173525] LustreError: 100255:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2580.191551] LustreError: 100255:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2580.199481] LustreError: 100255:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2580.318917] Lustre: server umount lustre-OST0000 complete [ 2580.325644] LustreError: 100255:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2589.675986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2589.899406] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2589.906604] LustreError: 98159:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2589.930926] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2589.935707] LustreError: 100720:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2589.943311] LustreError: 100720:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2589.947840] LustreError: 100720:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2590.006547] Lustre: server umount lustre-OST0000 complete [ 2590.008524] LustreError: 100720:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2600.584661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2600.788224] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2600.792334] Lustre: Found index 0 for lustre-OST0000, updating log [ 2600.797503] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2600.881403] Lustre: lustre-OST0000: new disk, initializing [ 2600.884282] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2600.888840] Lustre: 101257:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2601.251841] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2607.528919] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 2611.225594] Lustre: Failing over lustre-OST0000 [ 2611.486352] Lustre: server umount lustre-OST0000 complete [ 2616.485601] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2616.487531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2616.510361] Lustre: Skipped 2 previous similar messages [ 2621.920074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2621.928727] Lustre: Skipped 2 previous similar messages [ 2626.082972] Lustre: server umount lustre-MDT0000 complete [ 2627.044152] LustreError: 98173:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2627.055190] LustreError: 98173:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2629.386192] LustreError: 98154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788871970 with bad export cookie 8098744978293876113 [ 2629.387695] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2629.394804] LustreError: 98154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2629.725509] Lustre: server umount lustre-MDT0001 complete [ 2638.175220] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 08:52:57 (1788871977) [ 2656.229230] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 2666.801030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2667.071331] LustreError: 103602:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2667.139762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2671.927652] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 2680.812330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2686.872116] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 2690.297078] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2698.271810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2698.907563] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2706.091905] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2706.111526] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 2706.148288] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2709.108121] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 2711.537523] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 2711.581325] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2712.622061] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:03 [ 2712.654167] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2712.666257] Lustre: lustre-OST0000: Recovery over after 0:06, of 1 clients 1 recovered and 0 were evicted. [ 2712.671169] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2712.712851] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2720.203271] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2720.299405] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2720.458363] Lustre: lustre-OST0001: new disk, initializing [ 2720.461620] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2720.477835] Lustre: 105928:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2720.935499] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2720.945445] Lustre: Skipped 2 previous similar messages [ 2728.417326] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 2730.694048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2730.712315] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2730.883409] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2738.825172] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2748.830659] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2756.578702] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2756.591749] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2756.603081] Lustre: Skipped 2 previous similar messages [ 2756.620993] Lustre: Skipped 1 previous similar message [ 2759.895937] Lustre: server umount lustre-MDT0001 complete [ 2761.716951] LustreError: 103602:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2761.749817] LustreError: 103602:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2770.662307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2770.771583] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2770.778720] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2770.888564] Lustre: 103590:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2772.928523] LustreError: 107493:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2772.938963] LustreError: 107493:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2772.947308] LustreError: 103594:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2773.142839] LustreError: 107506:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.150@tcp: cfg command failed: rc = -17 [ 2773.153154] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2773.153154] [ 2776.511322] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 2777.079643] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2787.408318] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 08:55:26 (1788872126) [ 2810.100578] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 2831.383402] Lustre: 109542:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2858.556159] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2862.557883] Lustre: 110678:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2960.864683] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2960.876813] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2960.896156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2960.898671] Lustre: Skipped 1 previous similar message [ 2964.755904] Lustre: server umount lustre-MDT0000 complete [ 2966.520595] LustreError: 106666:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2966.550715] LustreError: 106666:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 2968.283419] LustreError: 105927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788872309 with bad export cookie 8098744978293877107 [ 2968.284554] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2968.293296] LustreError: 105927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2968.544834] Lustre: server umount lustre-MDT0001 complete [ 2982.799673] Lustre: server umount lustre-OST0000 complete [ 2996.998682] Lustre: server umount lustre-OST0001 complete [ 3004.064436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3018.138358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3030.325903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3042.192566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3063.194771] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 3077.062129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3077.082225] Lustre: lustre-MDT0000: reset Object Index mappings [ 3078.028663] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3078.034281] Lustre: Skipped 1 previous similar message [ 3083.188580] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 3094.496653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3094.533299] Lustre: lustre-MDT0001: reset Object Index mappings [ 3101.449361] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 3104.528546] Lustre: 115780:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3112.036026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3120.439527] LustreError: 116132:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3120.468818] LustreError: 116132:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 3120.490306] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 3122.499715] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:382 to 0x280000401:418) [ 3122.730239] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 3136.284540] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3142.665757] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:199 to 0x2c0000400:225) [ 3142.675089] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3145.954481] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 3155.390733] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3160.259271] Lustre: 117672:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3228.464229] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 09:02:46 (1788872566) [ 3253.747120] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3253.766796] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3253.779076] Lustre: Skipped 3 previous similar messages [ 3253.785426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3253.788058] Lustre: Skipped 4 previous similar messages [ 3257.685255] Lustre: server umount lustre-MDT0000 complete [ 3262.178225] LustreError: 114612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788872603 with bad export cookie 8098744978293924245 [ 3262.195121] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3262.206225] LustreError: 114612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3265.511299] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3265.536465] Lustre: Skipped 4 previous similar messages [ 3269.328865] Lustre: server umount lustre-MDT0001 complete [ 3280.843698] Lustre: server umount lustre-OST0000 complete [ 3292.040590] Lustre: server umount lustre-OST0001 complete [ 3301.774061] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_hostid [ 3314.955769] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 3365.861966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3366.170436] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3366.209035] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3366.358050] Lustre: lustre-MDT0000: new disk, initializing [ 3366.489982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3366.502041] Lustre: Skipped 3 previous similar messages [ 3366.529920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3372.292735] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 3379.945506] Lustre: server umount lustre-MDT0000 complete [ 3391.661391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3391.925238] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3391.938909] Lustre: 121946:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3391.955862] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3397.450389] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 3403.287693] Lustre: server umount lustre-MDT0000 complete [ 3413.892377] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 09:05:52 (1788872752) [ 3416.376476] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3419.013688] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 09:05:57 (1788872757) [ 3443.983640] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 3457.642040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3464.371806] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 3474.779541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3474.997398] Lustre: 124021:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3475.095171] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3475.102268] Lustre: Skipped 1 previous similar message [ 3475.236831] Lustre: lustre-MDT0001: new disk, initializing [ 3475.414672] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3475.418995] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3481.048821] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 3487.095935] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3494.718205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3495.062176] Lustre: lustre-OST0000: new disk, initializing [ 3495.065800] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3495.071744] Lustre: 125600:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3496.398888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3496.421308] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3496.509459] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3502.235214] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 3515.010993] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3515.176251] Lustre: lustre-OST0001: new disk, initializing [ 3515.179424] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3515.190976] Lustre: 126470:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3517.293236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3517.307027] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3517.521892] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3522.495644] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 3535.207519] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3540.116487] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3972.018468] Lustre: 132903:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3972.033815] Lustre: 132903:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3972.050461] Lustre: 132903:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3972.064530] Lustre: 132903:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3972.076446] Lustre: 132903:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3972.082597] Lustre: 132903:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4222.765840] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 09:19:21 (1788873561) [ 4238.305038] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4238.316758] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4238.330127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4238.333957] Lustre: Skipped 8 previous similar messages [ 4244.314662] Lustre: server umount lustre-MDT0000 complete [ 4244.455271] LustreError: 127289:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4244.469559] LustreError: 127289:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 17 previous similar messages [ 4247.716147] LustreError: 126469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788873588 with bad export cookie 8098744978293994399 [ 4247.717380] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4247.723100] LustreError: 126469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4249.572114] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4249.582722] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4249.590170] Lustre: Skipped 4 previous similar messages [ 4249.621531] Lustre: Skipped 5 previous similar messages [ 4257.704775] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4257.711768] Lustre: Skipped 3 previous similar messages [ 4262.367105] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4262.663606] Lustre: server umount lustre-MDT0001 complete [ 4280.287223] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4280.435588] Lustre: server umount lustre-OST0000 complete [ 4298.212683] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4298.487353] Lustre: server umount lustre-OST0001 complete [ 4306.028689] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_hostid [ 4319.988385] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 4360.758387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4361.036720] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4361.061837] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4361.118356] Lustre: lustre-MDT0000: new disk, initializing [ 4361.211994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4361.214949] Lustre: Skipped 5 previous similar messages [ 4361.229278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4366.307230] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 4377.185219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4377.307055] Lustre: 135954:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4377.358869] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4377.362347] Lustre: Skipped 1 previous similar message [ 4377.438922] Lustre: lustre-MDT0001: new disk, initializing [ 4377.586857] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4377.598982] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4382.175981] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 4391.765943] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4397.555850] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4404.754878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4405.060748] Lustre: lustre-OST0000: new disk, initializing [ 4405.064133] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4405.069547] Lustre: 137923:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4405.146121] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4405.153862] Lustre: Skipped 1 previous similar message [ 4406.286621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4406.297661] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4406.378658] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4411.239994] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 4422.697240] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4432.573681] Lustre: lustre-MDD0000: changelog on [ 4435.617754] Lustre: lustre-MDD0001: changelog on [ 6260.908965] Lustre: lustre-MDD0001: changelog off [ 6262.738578] Lustre: 137869:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6263.671671] Lustre: lustre-MDD0000: changelog off [ 6273.010489] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6273.014171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6273.022658] Lustre: Skipped 1 previous similar message [ 6273.032426] Lustre: Skipped 1 previous similar message [ 6278.120819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6278.131741] Lustre: Skipped 1 previous similar message [ 6283.249877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6283.255276] Lustre: Skipped 1 previous similar message [ 6284.256591] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6284.498561] Lustre: server umount lustre-OST0000 complete [ 6288.352824] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6288.364670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6294.676973] Lustre: server umount lustre-MDT0000 complete [ 6298.594170] LustreError: 135964:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6298.600804] LustreError: 135945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788875639 with bad export cookie 8098744978294773604 [ 6298.606157] LustreError: 135964:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 6298.606518] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6298.642384] LustreError: 135945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6299.032379] Lustre: server umount lustre-MDT0001 complete [ 6311.812335] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [ 6314.970604] Key type lgssc unregistered [ 6315.240847] LNet: 147040:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6315.245943] LNetError: 147040:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6316.263070] LNet: Removed LNI 192.168.201.150@tcp [ 6317.271282] Key type .llcrypt unregistered [ 6317.273298] Key type ._llcrypt unregistered [ 6336.877335] Key type ._llcrypt registered [ 6336.886190] Key type .llcrypt registered [ 6336.963377] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_hostid [ 6354.500677] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 6355.606278] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6355.626358] alg: No test for adler32 (adler32-zlib) [ 6356.872932] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 6357.218500] LNet: Added LNI 192.168.201.150@tcp [8/256/0/180] [ 6358.896688] Key type lgssc registered [ 6360.001423] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6397.011980] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6397.048799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6398.262381] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6398.285626] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6398.345794] Lustre: lustre-MDT0000: new disk, initializing [ 6398.468691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6398.493320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6402.373860] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 6411.753940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6411.911177] Lustre: 150252:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6411.950859] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6411.955051] Lustre: Skipped 1 previous similar message [ 6412.012362] Lustre: lustre-MDT0001: new disk, initializing [ 6412.088189] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6412.118566] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6412.131143] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6416.076433] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 6425.027709] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6431.264662] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6437.588993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6437.909491] Lustre: lustre-OST0000: new disk, initializing [ 6437.928863] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6437.938079] Lustre: 152219:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6438.055563] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6443.541880] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 6445.106776] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6445.118731] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6445.209745] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6452.499613] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6457.457072] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_module obdecho/obdecho [ 6458.995344] LustreError: 153221:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6459.000529] LustreError: 153221:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6460.528251] LustreError: 153317:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6471.139331] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6471.145236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6471.157836] Lustre: Skipped 2 previous similar messages [ 6471.176176] Lustre: Skipped 2 previous similar messages [ 6476.331959] Lustre: server umount lustre-MDT0000 complete [ 6479.429465] LustreError: 150242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788875820 with bad export cookie 10121257985057346551 [ 6479.433965] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6479.441686] LustreError: 150242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6479.783342] Lustre: server umount lustre-MDT0001 complete [ 6493.648540] Lustre: server umount lustre-OST0000 complete [ 6509.739704] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_hostid [ 6519.928122] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 6550.946893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6551.123357] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6551.149814] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6551.246889] Lustre: lustre-MDT0000: new disk, initializing [ 6551.340054] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6551.355833] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6555.569780] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 6565.259431] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6565.364680] Lustre: 157033:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6565.411189] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6565.414363] Lustre: Skipped 1 previous similar message [ 6565.471586] Lustre: lustre-MDT0001: new disk, initializing [ 6565.540595] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6565.582945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6565.605156] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6569.079260] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 6576.793540] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6581.768362] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6587.748284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6587.965562] Lustre: lustre-OST0000: new disk, initializing [ 6587.969935] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6587.979093] Lustre: 159000:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6588.065338] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6593.574545] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6593.580730] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6593.687857] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6594.036516] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 6602.312627] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6606.831811] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6607.048402] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6611.735714] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6611.960240] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6613.984839] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6613.991918] Lustre: Skipped 1 previous similar message [ 6613.992551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6614.002818] Lustre: Skipped 1 previous similar message [ 6619.111547] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6619.120662] Lustre: Skipped 1 previous similar message [ 6619.591395] Lustre: server umount lustre-OST0000 complete [ 6624.229612] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6624.236386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6624.250978] Lustre: Skipped 1 previous similar message [ 6628.673744] Lustre: server umount lustre-MDT0000 complete [ 6629.346017] LustreError: 157934:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6629.359536] LustreError: 157934:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 6631.971275] LustreError: 157023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788875972 with bad export cookie 10121257985057347517 [ 6631.974872] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6631.981802] LustreError: 157023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6632.302962] Lustre: server umount lustre-MDT0001 complete [ 6635.396648] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 09:59:34 (1788875974) [ 6636.627214] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6637.930243] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 09:59:37 (1788875977) [ 6644.412860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6644.722248] LustreError: 161409:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6644.774351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6648.228394] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 6649.836666] LustreError: 161410:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6654.956740] LustreError: 161409:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6655.243195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6659.114915] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 6666.004668] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6671.238847] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6677.346601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6677.583145] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6677.587346] Lustre: Skipped 1 previous similar message [ 6682.755829] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 6688.670368] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6700.518867] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6700.520329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6700.536184] Lustre: Skipped 2 previous similar messages [ 6700.543266] Lustre: Skipped 2 previous similar messages [ 6705.343222] Lustre: server umount lustre-MDT0000 complete [ 6705.636244] LustreError: 161409:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6705.654233] LustreError: 161409:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6708.332320] LustreError: 161391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788876049 with bad export cookie 10121257985057348399 [ 6708.343760] LustreError: 161391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6708.346669] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6709.652208] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.50@tcp (stopping) [ 6709.661624] Lustre: Skipped 2 previous similar messages [ 6710.754123] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6713.761214] LustreError: 161405:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6713.769156] LustreError: 161405:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6714.741194] Lustre: server umount lustre-MDT0001 complete [ 6720.756351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6732.726649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6732.981213] LustreError: 165262:0:(llog.c:1655:llog_backup()) MGC192.168.201.150@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6732.987888] Lustre: 165262:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.150@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6734.254108] LustreError: 165267:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6769.119817] LustreError: 165267:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6769.137831] LustreError: 165267:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 24 previous similar messages [ 6794.207995] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6794.215149] Lustre: 165262:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.150@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6833.632692] LustreError: 165268:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6833.655344] LustreError: 165268:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 39 previous similar messages [ 6835.206821] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6838.802664] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 6840.294249] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6845.832941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6853.612307] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x8c75ee833f6df488 [ 6853.621621] Lustre: MGC192.168.201.150@tcp: Connection restored to 0@lo (at 0@lo) [ 6857.324694] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 6859.241564] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6863.276936] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6864.735282] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6869.088501] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6870.516703] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6887.783061] Lustre: server umount lustre-MDT0000 complete [ 6889.956747] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6889.965575] Lustre: Skipped 1 previous similar message [ 6889.969437] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6890.458875] LustreError: 161391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788876231 with bad export cookie 10121257985057354888 [ 6890.464962] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6890.472826] LustreError: 161391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6890.703119] Lustre: server umount lustre-MDT0001 complete [ 6903.894454] Lustre: server umount lustre-OST0000 complete [ 6919.366206] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_hostid [ 6927.085724] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 6952.610721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6952.770642] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6952.786380] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6952.837803] Lustre: lustre-MDT0000: new disk, initializing [ 6952.890991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6955.891337] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 6963.577524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6963.676630] Lustre: 170362:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6963.812285] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6963.815794] Lustre: Skipped 2 previous similar messages [ 6963.844725] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6967.202918] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 6975.275968] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6979.742547] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6984.908075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6985.130924] Lustre: lustre-OST0000: new disk, initializing [ 6985.133516] Lustre: Skipped 1 previous similar message [ 6985.136509] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6985.139625] Lustre: Skipped 2 previous similar messages [ 6985.143214] Lustre: 172329:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6989.731432] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 6991.401353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6991.404046] Lustre: Skipped 1 previous similar message [ 6991.409703] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6991.504983] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6996.979704] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7000.393809] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7000.557795] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7004.834886] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7005.023612] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7006.689512] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7006.695852] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7006.704933] Lustre: Skipped 1 previous similar message [ 7006.712832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7006.720857] Lustre: Skipped 2 previous similar messages [ 7012.920384] Lustre: server umount lustre-OST0000 complete [ 7022.955288] Lustre: server umount lustre-MDT0000 complete [ 7026.323073] LustreError: 170355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788876367 with bad export cookie 10121257985057371702 [ 7026.332099] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7026.332273] LustreError: 170355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7026.687905] Lustre: server umount lustre-MDT0001 complete [ 7034.259295] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 10:06:13 (1788876373) [ 7040.179946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7040.473109] LustreError: 174740:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7040.485328] LustreError: 174740:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 22 previous similar messages [ 7044.266599] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 7052.412552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7056.086427] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 7062.453913] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7067.118397] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7073.183230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7078.148933] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 7084.263460] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7098.889870] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 10:07:18 (1788876438) [ 7103.462439] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7103.469912] LustreError: Skipped 1 previous similar message [ 7103.483305] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7103.505049] Lustre: Skipped 3 previous similar messages [ 7103.509632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7103.514182] Lustre: Skipped 8 previous similar messages [ 7106.532784] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7109.019927] Lustre: server umount lustre-OST0000 complete [ 7114.210990] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7118.600389] Lustre: server umount lustre-MDT0000 complete [ 7121.754552] LustreError: 174721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788876462 with bad export cookie 10121257985057372584 [ 7121.764366] LustreError: 174721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7121.768785] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7122.000958] Lustre: server umount lustre-MDT0001 complete [ 7131.283555] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [ 7133.564671] Key type lgssc unregistered [ 7133.858951] LNet: 178467:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7133.871473] LNetError: 178467:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7133.895461] LNet: Removed LNI 192.168.201.150@tcp [ 7134.563636] Key type .llcrypt unregistered [ 7134.567344] Key type ._llcrypt unregistered [ 7149.932308] Key type ._llcrypt registered [ 7149.934152] Key type .llcrypt registered [ 7150.031717] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 7151.127852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7151.164435] alg: No test for adler32 (adler32-zlib) [ 7152.145539] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 7152.326934] LNet: Added LNI 192.168.201.150@tcp [8/256/0/180] [ 7153.968791] Key type lgssc registered [ 7155.032861] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7161.230140] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7161.257214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7176.735453] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7181.855352] LustreError: 179707:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.150@tcp: failed processing log, type 4: rc = -110 [ 7186.976766] LustreError: 179707:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7186.988871] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7186.996678] LustreError: MGC192.168.201.150@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7187.009413] LustreError: 179707:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7187.013949] LustreError: 179707:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -22 [ 7187.017283] LustreError: 179707:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 7187.065799] Lustre: server umount lustre-OST0000 complete [ 7187.070855] LustreError: 179707:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7193.352228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7194.690161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7198.405974] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 7204.834037] LustreError: 180207:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7204.844052] LustreError: 180207:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7205.058440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7205.401169] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7209.352735] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 7215.532648] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7220.173947] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7225.302817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7225.454798] LustreError: 182024:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7225.465965] Lustre: 182024:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.201.150@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7225.593945] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7231.293235] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 7236.964600] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7241.184428] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7241.188463] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7241.195509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7241.199547] Lustre: Skipped 1 previous similar message [ 7241.210845] Lustre: Skipped 2 previous similar messages [ 7244.819793] Lustre: server umount lustre-MDT0000 complete [ 7246.312332] LustreError: 181648:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7246.336988] LustreError: 181648:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7247.603090] LustreError: 180188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788876588 with bad export cookie 4892219474356208737 [ 7247.604787] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7247.610776] LustreError: 180188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7247.809270] Lustre: server umount lustre-MDT0001 complete [ 7260.366483] Lustre: server umount lustre-OST0000 complete [ 7268.168619] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_hostid [ 7275.934973] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 7304.906468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7305.096193] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7305.115096] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7305.180173] Lustre: lustre-MDT0000: new disk, initializing [ 7305.248996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7305.269487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7309.063525] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 7318.124487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7318.217130] Lustre: 185544:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7318.268448] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7318.271924] Lustre: Skipped 1 previous similar message [ 7318.318408] Lustre: lustre-MDT0001: new disk, initializing [ 7318.379851] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7318.408580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7318.415162] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7321.940794] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 7329.450647] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7334.037706] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7339.919964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7340.199332] Lustre: lustre-OST0000: new disk, initializing [ 7340.203346] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7340.208577] Lustre: 187514:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7340.305635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7346.487418] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 7350.347192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7350.355769] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7350.503702] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7356.452071] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7360.420763] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7360.603295] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7364.691801] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7364.901947] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7371.234119] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7371.235263] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7371.247063] Lustre: Skipped 1 previous similar message [ 7371.249450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7371.271879] LustreError: Skipped 1 previous similar message [ 7372.644647] Lustre: server umount lustre-OST0000 complete [ 7376.354634] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7376.362930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7376.363887] Lustre: Skipped 2 previous similar messages [ 7376.373201] Lustre: Skipped 2 previous similar messages [ 7381.473159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7381.475569] Lustre: Skipped 1 previous similar message [ 7382.055573] Lustre: server umount lustre-MDT0000 complete [ 7385.484136] LustreError: 185538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788876726 with bad export cookie 4892219474356209444 [ 7385.487750] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7385.500422] LustreError: 185538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7385.770514] Lustre: server umount lustre-MDT0001 complete [ 7392.999767] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 10:12:12 (1788876732) [ 7406.885320] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [ 7409.157567] Key type lgssc unregistered [ 7409.418766] LNet: 190459:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7409.424758] LNetError: 190459:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7409.438942] LNet: Removed LNI 192.168.201.150@tcp [ 7410.171569] Key type .llcrypt unregistered [ 7410.176648] Key type ._llcrypt unregistered [ 7425.610593] Key type ._llcrypt registered [ 7425.614041] Key type .llcrypt registered [ 7425.723325] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 7426.539326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7426.656041] alg: No test for adler32 (adler32-zlib) [ 7427.687543] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 7427.914711] LNet: Added LNI 192.168.201.150@tcp [8/256/0/180] [ 7429.623431] Key type lgssc registered [ 7430.643309] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7437.722372] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7437.736181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7439.129192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7442.443732] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 7449.070902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7449.370854] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7452.569702] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 7458.948608] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7463.272526] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7469.484235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7469.697159] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7469.698896] LustreError: 193527:0:(llog.c:1696:llog_backup()) MGC192.168.201.150@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7469.703592] Lustre: 193527:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.201.150@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7469.823342] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7476.148746] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 7482.554556] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7485.413876] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7485.421371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7485.437479] Lustre: Skipped 1 previous similar message [ 7485.448951] Lustre: Skipped 2 previous similar messages [ 7490.531014] LustreError: 191661:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7490.550184] LustreError: 191661:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 7490.598077] Lustre: server umount lustre-MDT0000 complete [ 7493.690524] LustreError: 191647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788876834 with bad export cookie 33634938676465015 [ 7493.699946] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7493.703319] LustreError: 191647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7493.963957] Lustre: server umount lustre-MDT0001 complete [ 7506.313372] Lustre: server umount lustre-OST0000 complete [ 7513.606822] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_hostid [ 7521.943376] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 7548.376655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7548.568410] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7548.593194] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7548.644744] Lustre: lustre-MDT0000: new disk, initializing [ 7548.689590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7548.703571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7552.127149] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 7560.319155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7560.453317] Lustre: 197053:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7560.484121] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7560.486198] Lustre: Skipped 1 previous similar message [ 7560.488418] LustreError: 197835:0:(llog.c:1696:llog_backup()) MGC192.168.201.150@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7560.493327] LustreError: 197835:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7560.497314] Lustre: 197835:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.150@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7560.510914] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7560.513896] Lustre: Skipped 1 previous similar message [ 7560.562157] Lustre: lustre-MDT0001: new disk, initializing [ 7560.626549] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7560.658484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7560.672580] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7564.441348] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 7572.348307] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7576.493589] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7581.839096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7582.025823] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7582.027787] Lustre: Skipped 1 previous similar message [ 7582.030445] LustreError: 198996:0:(llog.c:1696:llog_backup()) MGC192.168.201.150@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7582.033628] LustreError: 198996:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7582.037409] Lustre: 198996:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.150@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7582.042762] Lustre: 198996:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7582.067674] Lustre: lustre-OST0000: new disk, initializing [ 7582.073504] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7582.078402] Lustre: 199019:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7582.162576] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7586.378286] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 7591.969293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7591.978552] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7592.021302] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7592.635374] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7596.049944] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7596.208480] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7599.078320] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7599.209359] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7602.147682] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7602.148055] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7602.159499] LustreError: Skipped 1 previous similar message [ 7602.168425] Lustre: Skipped 1 previous similar message [ 7602.170322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7606.778134] Lustre: server umount lustre-OST0000 complete [ 7611.872534] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7611.881636] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7611.903636] Lustre: Skipped 1 previous similar message [ 7611.908221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7611.910646] Lustre: Skipped 1 previous similar message [ 7617.429422] Lustre: server umount lustre-MDT0000 complete [ 7617.508488] LustreError: 197060:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7617.516479] LustreError: 197060:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7619.634954] LustreError: 197045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788876960 with bad export cookie 33634938676465729 [ 7619.637179] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7619.644877] LustreError: 197045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7619.855513] Lustre: server umount lustre-MDT0001 complete [ 7625.626754] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 10:16:05 (1788876965) [ 7637.091374] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 7644.610404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7644.828852] LustreError: 202144:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7644.864761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7647.796483] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 7650.276054] LustreError: 202145:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7653.965254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7657.439763] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 7659.698337] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7664.656162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7669.852422] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 7676.233272] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7676.365688] Lustre: lustre-OST0001: new disk, initializing [ 7676.368621] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7676.372284] Lustre: 204450:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7680.700604] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 7683.139463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7683.144948] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7683.196651] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7688.321735] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7694.333492] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7695.747325] Lustre: DEBUG MARKER: ADD OST3 [ 7699.084505] loop: module loaded [ 7699.146032] loop0: detected capacity change from 0 to 204800000 [ 7699.288369] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7699.295210] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7699.301080] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7703.640870] loop0: detected capacity change from 0 to 204800000 [ 7705.400456] Lustre: DEBUG MARKER: STOP OST3 [ 7707.900347] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7708.016953] Lustre: lustre-OST0003: new disk, initializing [ 7708.021433] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7708.026632] Lustre: 206377:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7708.107578] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7708.109921] Lustre: Skipped 3 previous similar messages [ 7709.837051] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7709.865023] Lustre: Failing over lustre-OST0003 [ 7709.870100] Lustre: lustre-OST0003: Not available for connect from 0@lo (stopping) [ 7709.875899] Lustre: Skipped 2 previous similar messages [ 7709.972907] Lustre: server umount lustre-OST0003 complete [ 7711.416340] LustreError: 204930:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.201.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7711.909073] LustreError: 206583:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7712.940720] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 7715.316052] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7715.325148] Lustre: Skipped 1 previous similar message [ 7719.787962] LustreError: 203630:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.201.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7719.797105] LustreError: 203630:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 7720.722931] Lustre: DEBUG MARKER: START OST3 again [ 7725.446462] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7725.646773] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7727.094928] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7727.120451] Lustre: lustre-OST0003: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 7727.121919] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7727.135101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7727.142168] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7730.273965] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 7730.707274] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000401 [ 7747.575545] Lustre: Failing over lustre-OST0003 [ 7747.649915] Lustre: server umount lustre-OST0003 complete [ 7751.145275] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7751.145486] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7751.149982] LustreError: 203630:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7751.164808] Lustre: Skipped 1 previous similar message [ 7751.175291] LustreError: 203630:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 7766.499761] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7766.504445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7766.514177] Lustre: Skipped 3 previous similar messages [ 7766.518112] Lustre: Skipped 2 previous similar messages [ 7776.743533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7776.759053] Lustre: Skipped 8 previous similar messages [ 7777.247136] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7777.382352] Lustre: server umount lustre-MDT0000 complete [ 7780.517049] LustreError: 204449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788877121 with bad export cookie 33634938676466611 [ 7780.524972] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7780.529732] LustreError: 204449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7785.894538] LustreError: 202141:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7785.905374] LustreError: 202141:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 7786.871735] Lustre: server umount lustre-MDT0001 complete [ 7789.654869] Lustre: server umount lustre-OST0000 complete [ 7792.307550] Lustre: server umount lustre-OST0001 complete [ 7796.959826] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_hostid [ 7802.988156] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 7828.235942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7828.357542] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7828.377323] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7828.430924] Lustre: lustre-MDT0000: new disk, initializing [ 7828.474920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7828.477018] Lustre: Skipped 1 previous similar message [ 7828.484283] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7831.116193] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 7838.514906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7838.612955] Lustre: 211907:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7838.819188] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7841.924228] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 7848.200577] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7851.890613] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7856.771922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7856.982910] Lustre: lustre-OST0000: new disk, initializing [ 7856.985795] Lustre: Skipped 1 previous similar message [ 7856.988330] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7856.993165] Lustre: Skipped 2 previous similar messages [ 7856.997623] Lustre: 213873:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7858.208737] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7858.214177] Lustre: Skipped 1 previous similar message [ 7858.223377] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7858.320951] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7860.922355] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 7866.901883] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7870.071355] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7870.194341] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7872.584848] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7872.737590] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7874.017273] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7874.024499] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7874.033236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7874.035291] Lustre: Skipped 3 previous similar messages [ 7874.042379] Lustre: Skipped 4 previous similar messages [ 7888.351163] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7888.477614] Lustre: server umount lustre-OST0000 complete [ 7896.924628] Lustre: server umount lustre-MDT0000 complete [ 7898.926945] LustreError: 211899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788877239 with bad export cookie 33634938676468900 [ 7898.928062] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7898.933547] LustreError: 211899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7899.136475] Lustre: server umount lustre-MDT0001 complete [ 7900.976203] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 10:20:40 (1788877240) [ 7909.247758] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_hostid [ 7914.462323] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 7933.672249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7933.819469] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7933.841364] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7933.897237] Lustre: lustre-MDT0000: new disk, initializing [ 7933.980483] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7936.370390] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 7942.731926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7942.823210] Lustre: 218476:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7942.999507] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7945.566897] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 7951.400834] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7954.423367] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7958.363725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7958.493499] Lustre: 220442:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7958.530701] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7958.533754] Lustre: Skipped 4 previous similar messages [ 7960.407413] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7961.863532] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 7967.531388] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7970.312374] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7970.409088] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7972.745697] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7972.854938] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7975.910460] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7975.911717] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7975.913069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7975.913075] Lustre: Skipped 6 previous similar messages [ 7975.915860] LustreError: Skipped 1 previous similar message [ 7975.930495] Lustre: Skipped 3 previous similar messages [ 7988.192059] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7988.310385] Lustre: server umount lustre-OST0000 complete [ 7996.669291] Lustre: server umount lustre-MDT0000 complete [ 7998.416462] LustreError: 220441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788877339 with bad export cookie 33634938676469782 [ 7998.421242] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7998.424099] LustreError: 220441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7998.601641] Lustre: server umount lustre-MDT0001 complete [ 8003.341753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8003.589850] LustreError: 222657:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8006.202669] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 8011.576415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8014.320066] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 8018.628444] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8021.743859] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8026.103419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8029.598673] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 8033.956585] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8044.000527] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8044.005149] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8044.010123] Lustre: Skipped 2 previous similar messages [ 8044.012400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8044.014941] Lustre: Skipped 9 previous similar messages [ 8056.287121] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8056.391667] Lustre: server umount lustre-OST0000 complete [ 8064.766831] Lustre: server umount lustre-MDT0000 complete [ 8067.049337] LustreError: 222638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788877408 with bad export cookie 33634938676470664 [ 8067.053671] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8067.058947] LustreError: 222638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8067.292802] Lustre: server umount lustre-MDT0001 complete [ 8170.677638] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [ 8173.245799] Key type lgssc unregistered [ 8173.600497] LNet: 226316:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8173.606431] LNetError: 226316:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8173.635550] LNet: Removed LNI 192.168.201.150@tcp [ 8174.428239] Key type .llcrypt unregistered [ 8174.430103] Key type ._llcrypt unregistered [ 8205.768852] Key type ._llcrypt registered [ 8205.776241] Key type .llcrypt registered [ 8206.024423] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_hostid [ 8225.862176] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 8226.986767] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8227.011792] alg: No test for adler32 (adler32-zlib) [ 8227.952251] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 8228.161872] LNet: Added LNI 192.168.201.150@tcp [8/256/0/180] [ 8229.807245] Key type lgssc registered [ 8231.473189] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8304.688514] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8304.712547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8306.192629] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8306.316361] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8306.509770] Lustre: lustre-MDT0000: new disk, initializing [ 8306.655699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8306.672981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8312.381645] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 8324.426732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8324.541792] Lustre: 229673:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8324.598136] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8324.602580] Lustre: Skipped 1 previous similar message [ 8324.694363] Lustre: lustre-MDT0001: new disk, initializing [ 8324.809605] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8324.854159] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8324.865120] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8329.869186] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 8339.734359] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8346.366461] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8353.035383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8353.299529] Lustre: lustre-OST0000: new disk, initializing [ 8353.302975] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8353.309067] Lustre: 231642:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8353.440345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8360.636873] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 8362.100150] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8362.109555] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8362.208514] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8371.301089] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8376.247085] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8376.439995] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8381.290772] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8381.510676] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8388.074556] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8388.080164] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8388.080941] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8388.094964] Lustre: Skipped 1 previous similar message [ 8388.128908] Lustre: Skipped 1 previous similar message [ 8390.081277] Lustre: server umount lustre-OST0000 complete [ 8396.771801] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8396.775922] LustreError: Skipped 1 previous similar message [ 8396.778246] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8396.785851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8398.310220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8398.310803] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8400.998539] Lustre: server umount lustre-MDT0000 complete [ 8405.590183] LustreError: 229665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788877746 with bad export cookie 16671755748876916205 [ 8405.598102] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8405.907662] Lustre: server umount lustre-MDT0001 complete [ 8427.624968] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 8438.014532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8438.261443] LustreError: 234578:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8438.284024] LustreError: 234578:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8438.310161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8443.004665] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 8443.360831] LustreError: 234579:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8448.481158] LustreError: 234578:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8452.218338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8452.656361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8457.817490] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 8461.449875] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8468.947119] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8469.363030] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8476.716115] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 8486.243279] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8486.436113] Lustre: lustre-OST0001: new disk, initializing [ 8486.440319] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8486.447203] Lustre: 236885:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8486.546107] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8494.553799] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 8496.724934] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8496.735369] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8496.860970] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8506.677316] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8515.906880] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8530.429699] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 10:31:09 (1788877869) [ 8532.966118] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8532.977903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8532.981859] Lustre: Skipped 4 previous similar messages [ 8538.115831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8538.135930] Lustre: Skipped 3 previous similar messages [ 8540.769322] Lustre: server umount lustre-MDT0000 complete [ 8543.207693] LustreError: 234579:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8543.240324] LustreError: 234579:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8544.698327] LustreError: 234559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788877885 with bad export cookie 16671755748876917087 [ 8544.709974] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8544.712931] LustreError: 234559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8545.072772] Lustre: server umount lustre-MDT0001 complete [ 8560.101167] Lustre: server umount lustre-OST0000 complete [ 8564.196832] Lustre: 227894:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788877889/real 1788877889] req@ffff967fff3b5f80 x1875774085907712/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788877905 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8564.228823] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8564.245761] Lustre: Skipped 3 previous similar messages [ 8565.116615] Lustre: server umount lustre-OST0001 complete [ 8574.768481] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_hostid [ 8587.210537] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 8623.831409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8624.070682] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8624.174693] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8624.283413] Lustre: lustre-MDT0000: new disk, initializing [ 8624.386710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8624.405732] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8629.176959] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 8640.085655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8640.210698] Lustre: 240835:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8640.252423] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8640.255784] Lustre: Skipped 1 previous similar message [ 8640.317639] Lustre: lustre-MDT0001: new disk, initializing [ 8640.523611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8640.534250] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8645.245356] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 8654.846948] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8660.381707] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8666.597607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8666.880925] Lustre: lustre-OST0000: new disk, initializing [ 8666.886884] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8666.894231] Lustre: 242801:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8668.506512] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8668.525949] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8668.600109] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8672.813402] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 8681.536483] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8686.035168] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8686.310909] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8691.146197] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8691.355154] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8694.239973] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8694.246472] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8694.251187] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8694.253641] Lustre: Skipped 1 previous similar message [ 8694.270591] Lustre: Skipped 1 previous similar message [ 8699.197301] Lustre: server umount lustre-OST0000 complete [ 8707.041745] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8707.055192] LustreError: Skipped 1 previous similar message [ 8712.674449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8712.684140] Lustre: Skipped 5 previous similar messages [ 8717.279136] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8717.452157] Lustre: server umount lustre-MDT0000 complete [ 8717.797568] LustreError: 240845:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8717.815115] LustreError: 240845:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8720.707493] LustreError: 242800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788878061 with bad export cookie 16671755748876919222 [ 8720.716989] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8720.721764] LustreError: 242800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8721.168352] Lustre: server umount lustre-MDT0001 complete [ 8728.855481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8729.236259] LustreError: 245021:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8729.300562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8729.302585] Lustre: Skipped 2 previous similar messages [ 8734.674961] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 8743.352978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8748.221616] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 8756.165309] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8762.525965] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8769.398204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8776.381498] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 8784.332915] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8928.736101] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8928.744646] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8928.749857] Lustre: Skipped 2 previous similar messages [ 8928.753046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8942.047137] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8942.140553] Lustre: server umount lustre-OST0000 complete [ 8950.113879] Lustre: server umount lustre-MDT0000 complete [ 8951.624854] LustreError: 245002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788878292 with bad export cookie 16671755748876920104 [ 8951.625807] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8951.628492] LustreError: 245002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8951.758419] Lustre: server umount lustre-MDT0001 complete [ 9048.757486] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [ 9049.869641] Key type lgssc unregistered [ 9050.005388] LNet: 248626:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9050.007751] LNetError: 248626:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9050.017387] LNet: Removed LNI 192.168.201.150@tcp [ 9050.348122] Key type .llcrypt unregistered [ 9050.349426] Key type ._llcrypt unregistered [ 9059.212645] Key type ._llcrypt registered [ 9059.213847] Key type .llcrypt registered [ 9059.260367] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_hostid [ 9066.721655] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 9067.161091] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9067.171375] alg: No test for adler32 (adler32-zlib) [ 9068.035122] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 9068.134430] LNet: Added LNI 192.168.201.150@tcp [8/256/0/180] [ 9069.729860] Key type lgssc registered [ 9070.149433] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9088.427880] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 9092.684824] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9092.698344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9093.794543] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9093.807843] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9093.843898] Lustre: lustre-MDT0000: new disk, initializing [ 9093.878113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9093.885431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9095.548702] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 9100.656078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9100.705691] Lustre: 252665:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9100.735742] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9100.738418] Lustre: Skipped 1 previous similar message [ 9100.779236] Lustre: lustre-MDT0001: new disk, initializing [ 9100.821466] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9100.836497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9100.841626] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9102.512903] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 9105.002255] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9107.559646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9107.674654] Lustre: lustre-OST0000: new disk, initializing [ 9107.677443] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9107.683074] Lustre: 254297:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9107.731821] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9109.998538] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 9113.597055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9113.603648] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9113.660845] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9114.699672] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9114.776787] Lustre: lustre-OST0001: new disk, initializing [ 9114.778658] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9114.782522] Lustre: 255167:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9114.829660] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9117.130753] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 9121.812349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9121.817115] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9121.866695] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9123.223914] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9126.330288] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9131.488536] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9131.491781] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9131.496925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9132.002580] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9132.004875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9132.008863] Lustre: Skipped 2 previous similar messages [ 9132.011792] Lustre: Skipped 1 previous similar message [ 9135.014279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9135.019226] Lustre: Skipped 3 previous similar messages [ 9136.811690] Lustre: server umount lustre-MDT0000 complete [ 9138.548499] LustreError: 252658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788878479 with bad export cookie 11772563578923217977 [ 9138.552267] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9138.553826] LustreError: 252658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9138.745871] Lustre: server umount lustre-MDT0001 complete [ 9150.942058] Lustre: server umount lustre-OST0000 complete [ 9162.919518] Lustre: server umount lustre-OST0001 complete [ 9169.188336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9172.964341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9176.293122] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9180.868441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9189.458325] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 9194.079321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9194.090621] Lustre: lustre-MDT0000: reset Object Index mappings [ 9194.358177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9196.074554] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 9199.585927] LustreError: 260166:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9199.592387] LustreError: 260166:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9199.745589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9199.755877] Lustre: lustre-MDT0001: reset Object Index mappings [ 9201.690477] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 9202.964099] Lustre: 261304:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9205.841469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9206.011506] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9206.015030] Lustre: Skipped 1 previous similar message [ 9208.646873] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 9209.062108] LustreError: 261658:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9212.567642] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9215.497071] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 9219.050157] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9219.050411] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9224.125114] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9225.694320] Lustre: 263176:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9229.283282] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9229.287587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9229.291088] Lustre: Skipped 1 previous similar message [ 9229.293669] Lustre: Skipped 1 previous similar message [ 9233.204379] Lustre: server umount lustre-MDT0000 complete [ 9234.400845] LustreError: 260166:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9234.408035] LustreError: 260166:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9234.763330] LustreError: 260146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788878575 with bad export cookie 11772563578923224711 [ 9234.767674] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9234.768027] LustreError: 260146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9234.885344] Lustre: server umount lustre-MDT0001 complete [ 9246.734897] Lustre: server umount lustre-OST0000 complete [ 9258.527888] Lustre: server umount lustre-OST0001 complete [ 9266.479489] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 10:43:26 (1788878606) [ 9274.093581] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_hostid [ 9278.887633] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 9294.927287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9295.022766] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9295.037483] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9295.078773] Lustre: lustre-MDT0000: new disk, initializing [ 9295.111350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9295.114916] Lustre: Skipped 1 previous similar message [ 9295.120435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9297.064589] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 9302.226046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9302.282744] Lustre: 267421:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9302.415569] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9304.087417] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 9308.428395] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9310.593344] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9313.177376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9313.291981] Lustre: lustre-OST0000: new disk, initializing [ 9313.293744] Lustre: Skipped 1 previous similar message [ 9313.295837] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9313.299411] Lustre: Skipped 2 previous similar messages [ 9313.302143] Lustre: 269389:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9315.203646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9315.207188] Lustre: Skipped 1 previous similar message [ 9315.209538] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9315.240757] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9315.577150] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 9320.081709] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9322.343400] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9322.447353] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9324.138154] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9324.236698] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9325.538146] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9325.538781] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9325.539353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9325.539357] Lustre: Skipped 2 previous similar messages [ 9325.543401] Lustre: Skipped 2 previous similar messages [ 9325.546545] LustreError: Skipped 1 previous similar message [ 9335.781481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9335.785237] Lustre: Skipped 4 previous similar messages [ 9339.359147] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9339.420973] Lustre: server umount lustre-OST0000 complete [ 9340.899211] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9340.903972] Lustre: Skipped 1 previous similar message [ 9355.232068] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9355.306898] Lustre: server umount lustre-MDT0000 complete [ 9356.256128] LustreError: 268225:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9356.264713] LustreError: 268225:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9356.942376] LustreError: 267415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788878697 with bad export cookie 11772563578923226783 [ 9356.944905] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9356.950247] LustreError: 267415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9357.079225] Lustre: server umount lustre-MDT0001 complete [ 9364.299249] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 9367.903690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9368.035567] LustreError: 272324:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9368.053231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9368.056067] Lustre: Skipped 2 previous similar messages [ 9369.584419] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 9372.556639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9373.961864] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 9374.941084] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9377.254872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9379.547945] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 9382.782645] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9382.861783] Lustre: lustre-OST0001: new disk, initializing [ 9382.863874] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9382.867416] Lustre: 274633:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9384.648097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9384.651911] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9384.692091] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9385.069312] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 9389.818586] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9391.156944] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9393.073801] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 9393.079458] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 9398.240471] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9398.242694] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9398.248180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9398.250258] Lustre: Skipped 6 previous similar messages [ 9402.386676] Lustre: server umount lustre-MDT0000 complete [ 9404.023177] LustreError: 274632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788878744 with bad export cookie 11772563578923227665 [ 9404.026335] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9404.030598] LustreError: 274632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9404.148849] Lustre: server umount lustre-MDT0001 complete [ 9415.761080] Lustre: server umount lustre-OST0000 complete [ 9427.770349] Lustre: server umount lustre-OST0001 complete [ 9433.884370] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 9437.976716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9438.144208] LustreError: 277971:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9438.177614] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9438.181207] Lustre: Skipped 3 previous similar messages [ 9439.848869] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 9443.345249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9445.131658] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 9446.279293] Lustre: 279111:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9449.199708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9452.014837] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 9455.879336] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9458.736976] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 9463.272420] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 9463.284819] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 9467.785179] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9469.071319] Lustre: 280989:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9470.886421] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 9470.889636] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 9475.231967] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 10:46:55 (1788878815) [ 9488.865552] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9488.867336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9488.870169] Lustre: Skipped 5 previous similar messages [ 9488.872579] Lustre: Skipped 5 previous similar messages [ 9492.891935] Lustre: server umount lustre-MDT0000 complete [ 9493.984381] LustreError: 277989:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9493.990491] LustreError: 277989:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 9494.202510] LustreError: 277951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788878835 with bad export cookie 11772563578923234035 [ 9494.206310] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9494.207523] LustreError: 277951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9500.403239] Lustre: server umount lustre-MDT0001 complete [ 9515.999124] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9516.035851] Lustre: server umount lustre-OST0000 complete [ 9523.572713] Lustre: server umount lustre-OST0001 complete [ 9525.962391] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_hostid [ 9529.583036] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 9545.517660] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 9548.926792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9549.014500] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9549.025327] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9549.058911] Lustre: lustre-MDT0000: new disk, initializing [ 9549.086392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9550.453686] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 9554.626935] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9554.666984] Lustre: 285510:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9554.764867] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9556.125650] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 9560.766981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9560.883590] Lustre: 287142:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9561.990661] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9563.142668] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 9567.638751] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9567.734720] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9567.737795] Lustre: Skipped 6 previous similar messages [ 9568.866948] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9569.808149] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 9574.519899] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9580.247247] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 10:48:40 (1788878920) [ 9584.100351] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9584.101310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9584.104097] Lustre: Skipped 5 previous similar messages [ 9584.105805] Lustre: Skipped 3 previous similar messages [ 9587.100772] Lustre: server umount lustre-MDT0000 complete [ 9588.390232] LustreError: 286305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788878929 with bad export cookie 11772563578923241966 [ 9588.393854] LustreError: 286305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9588.394057] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9588.510568] Lustre: server umount lustre-MDT0001 complete [ 9600.071181] Lustre: server umount lustre-OST0000 complete [ 9611.556728] Lustre: server umount lustre-OST0001 complete [ 9614.555938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9614.723304] LustreError: 290257:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9614.729670] LustreError: 290257:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9616.211214] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 9619.424932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9621.075904] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 9623.859350] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9625.984315] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9628.714689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9631.108346] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 9633.757761] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9650.867244] loop1: detected capacity change from 0 to 204800000 [ 9650.946188] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9650.951387] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9650.955915] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9652.659770] loop1: detected capacity change from 0 to 204800000 [ 9654.603922] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9654.641699] LustreError: 290246:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs26367: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 9654.648105] LustreError: 290246:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs26367 [ 9654.650446] LustreError: 290246:0:(mgs_handler.c:517:mgs_target_reg()) fs26367-MDT0001: Error checking target index: rc = -13 [ 9654.654043] LustreError: fs26367-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 9654.658253] LustreError: 293993:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [ 9654.660621] LustreError: 293993:0:(tgt_mount.c:2128:server_put_super()) no obd fs26367-MDT0001 [ 9654.662529] LustreError: 293993:0:(tgt_mount.c:116:server_deregister_mount()) fs26367-MDT0001 not registered [ 9654.674274] Lustre: server umount fs26367-MDT0001 complete [ 9654.676131] LustreError: 293993:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 9657.808534] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9657.840697] Lustre: Setting parameter fs26367-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs26367-MDT0001 [ 9657.844767] Lustre: Skipped 2 previous similar messages [ 9657.861297] Lustre: srv-fs26367-MDT0001: No data found on store. Initialize space. [ 9657.863573] Lustre: Skipped 4 previous similar messages [ 9657.898972] Lustre: fs26367-MDT0001: new disk, initializing [ 9657.900809] Lustre: Skipped 3 previous similar messages [ 9657.941316] Lustre: srv-fs26367-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 9657.944505] Lustre: 294457:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs26367-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9657.948032] LustreError: 294457:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs26367-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9659.333160] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 9662.232125] Lustre: server umount fs26367-MDT0001 complete [ 9666.405686] loop1: detected capacity change from 0 to 204800000 [ 9668.276753] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9668.379381] Lustre: 296027:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs26367-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9668.382567] LustreError: 296027:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs26367-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9669.755677] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 9671.318948] Lustre: server umount fs26367-MDT0001 complete [ 9675.233274] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9675.238422] Lustre: Skipped 3 previous similar messages [ 9680.234857] LustreError: 292082:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9680.241457] LustreError: 292082:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 24 previous similar messages [ 9680.600666] Lustre: server umount lustre-MDT0000 complete [ 9683.413212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9683.452235] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9684.872455] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 9687.423619] loop1: detected capacity change from 0 to 204800000 [ 9689.063825] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9689.312243] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9689.423663] Lustre: 298469:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs26367-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9689.426735] LustreError: 298469:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs26367-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9690.796411] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 9692.437558] Lustre: server umount fs26367-MDT0001 complete [ 9705.239064] Lustre: server umount lustre-OST0000 complete [ 9709.536679] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9712.668446] Lustre: server umount lustre-MDT0000 complete [ 9714.023055] LustreError: 290237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788879054 with bad export cookie 11772563578923244640 [ 9714.027044] LustreError: 290237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9714.149541] Lustre: server umount lustre-MDT0001 complete [ 9717.744825] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [ 9718.789412] Key type lgssc unregistered [ 9718.921395] LNet: 300806:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9718.925119] LNetError: 300806:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9718.933299] LNet: Removed LNI 192.168.201.150@tcp [ 9719.276159] Key type .llcrypt unregistered [ 9719.277570] Key type ._llcrypt unregistered [ 9727.183795] Key type ._llcrypt registered [ 9727.185266] Key type .llcrypt registered [ 9727.226871] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 9727.618463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9727.658825] alg: No test for adler32 (adler32-zlib) [ 9728.518754] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 9728.616252] LNet: Added LNI 192.168.201.150@tcp [8/256/0/180] [ 9730.199168] Key type lgssc registered [ 9730.545400] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9733.072059] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9733.078205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9734.243780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9735.540745] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 9738.473848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9738.624193] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9739.972251] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 9742.457482] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9744.344602] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9746.625498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9746.748188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9748.674490] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 9748.835637] LustreError: 303877:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9748.842469] LustreError: 303877:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9751.119630] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9754.084780] LustreError: 303877:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9759.206630] LustreError: 303877:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9759.211921] LustreError: 303877:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9761.648894] LustreError: 303877:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9766.763672] LustreError: 304261:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9766.771363] LustreError: 304261:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9767.749183] loop1: detected capacity change from 0 to 204800000 [ 9767.809585] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9767.812983] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9767.816577] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9769.599037] loop1: detected capacity change from 0 to 204800000 [ 9771.528986] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9771.579355] Lustre: fs30803-OST0000: new disk, initializing [ 9771.580782] Lustre: srv-fs30803-OST0000: No data found on store. Initialize space. [ 9771.583332] Lustre: 305802:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9771.608554] Lustre: fs30803-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9773.554051] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 9776.994666] Lustre: server umount fs30803-OST0000 complete [ 9777.003477] LustreError: 303877:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9777.008615] LustreError: 303877:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 9781.017770] loop1: detected capacity change from 0 to 204800000 [ 9782.863806] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9782.919423] Lustre: fs30803-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9784.774949] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 9787.564889] loop2: detected capacity change from 0 to 204800000 [ 9787.629295] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9787.633190] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9787.636738] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9789.275892] loop2: detected capacity change from 0 to 204800000 [ 9791.061718] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9791.088503] LustreError: fs30803-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 9791.092734] LustreError: fs30803-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 9791.095409] LustreError: 308839:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [ 9791.097613] LustreError: 308839:0:(tgt_mount.c:2128:server_put_super()) no obd fs30803-OST0001 [ 9791.099488] LustreError: 308839:0:(tgt_mount.c:116:server_deregister_mount()) fs30803-OST0001 not registered [ 9791.108377] Lustre: server umount fs30803-OST0001 complete [ 9791.109580] LustreError: 308839:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 9794.747531] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9794.795471] Lustre: fs30803-OST0001: new disk, initializing [ 9794.797070] Lustre: srv-fs30803-OST0001: No data found on store. Initialize space. [ 9794.799284] Lustre: 309389:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9794.823788] Lustre: fs30803-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9795.043495] LustreError: 303877:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9795.047022] LustreError: 303877:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 9796.780128] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 9800.268925] Lustre: server umount fs30803-OST0000 complete [ 9803.317769] Lustre: server umount fs30803-OST0001 complete [ 9807.379086] loop1: detected capacity change from 0 to 204800000 [ 9809.314554] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9811.263876] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 9813.424234] loop2: detected capacity change from 0 to 204800000 [ 9815.217110] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9815.279792] Lustre: fs30803-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9815.281931] Lustre: Skipped 1 previous similar message [ 9817.234429] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 9820.150651] Lustre: server umount fs30803-OST0000 complete [ 9823.205764] Lustre: server umount fs30803-OST0001 complete [ 9830.880786] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9830.882373] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9830.882590] LustreError: 303878:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9830.882599] LustreError: 303878:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 18 previous similar messages [ 9830.884438] LustreError: Skipped 1 previous similar message [ 9830.890444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9830.904294] Lustre: Skipped 1 previous similar message [ 9833.047151] Lustre: server umount lustre-OST0000 complete [ 9836.001236] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9836.001958] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9836.002455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9836.005068] Lustre: Skipped 2 previous similar messages [ 9840.472467] Lustre: server umount lustre-MDT0000 complete [ 9841.803204] LustreError: 302079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788879182 with bad export cookie 13543855926012701748 [ 9841.806109] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9841.807507] LustreError: 302079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9841.920759] Lustre: server umount lustre-MDT0001 complete [ 9845.684270] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [ 9846.723554] Key type lgssc unregistered [ 9846.849364] LNet: 315296:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9846.851758] LNetError: 315296:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9846.859300] LNet: Removed LNI 192.168.201.150@tcp [ 9847.193121] Key type .llcrypt unregistered [ 9847.194338] Key type ._llcrypt unregistered [ 9857.275990] Key type ._llcrypt registered [ 9857.277511] Key type .llcrypt registered [ 9857.315598] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_hostid [ 9863.706994] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [ 9864.024111] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9864.031674] alg: No test for adler32 (adler32-zlib) [ 9864.883947] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [ 9864.968750] LNet: Added LNI 192.168.201.150@tcp [8/256/0/180] [ 9866.551169] Key type lgssc registered [ 9866.878885] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9878.416225] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9878.421049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9879.498378] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9879.508367] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9879.542186] Lustre: lustre-MDT0000: new disk, initializing [ 9879.564062] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9879.570572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9880.908740] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 9885.153050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9885.191437] Lustre: 319119:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9885.213379] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9885.216190] Lustre: Skipped 1 previous similar message [ 9885.254657] Lustre: lustre-MDT0001: new disk, initializing [ 9885.292593] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9885.306564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9885.309441] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9886.780817] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 9890.581029] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9892.458210] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9894.785352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9894.907519] Lustre: lustre-OST0000: new disk, initializing [ 9894.908915] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9894.911104] Lustre: 321087:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9894.942745] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9896.845323] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [ 9900.457929] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9902.836468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9902.839736] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9902.866537] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9908.209066] Lustre: Mounted lustre-client - version 2.17.58_39_g331ba7d [ 9908.526516] Lustre: Unmounted lustre-client [ 9913.313405] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9913.313909] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9913.315848] LustreError: Skipped 1 previous similar message [ 9913.319689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9913.321946] Lustre: Skipped 1 previous similar message [ 9917.453719] Lustre: server umount lustre-OST0000 complete [ 9921.503799] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9921.505980] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9921.509009] Lustre: Skipped 1 previous similar message [ 9921.510535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9923.552166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9923.552515] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9924.817991] Lustre: server umount lustre-MDT0000 complete [ 9926.019170] LustreError: 319114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788879266 with bad export cookie 2806368827196215304 [ 9926.021647] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9926.023776] LustreError: 319114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9926.143321] Lustre: server umount lustre-MDT0001 complete [ 9929.868615] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [ 9930.907173] Key type lgssc unregistered [ 9956.639344] LNet: 323245:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9956.641747] LNetError: 323245:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10032.421352] LNet: Removed LNI 192.168.201.150@tcp [10032.729633] Key type .llcrypt unregistered [10032.730632] Key type ._llcrypt unregistered [10038.348193] Key type ._llcrypt registered [10038.349261] Key type .llcrypt registered [10038.386189] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_hostid [10044.790721] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [10045.066934] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10045.077431] alg: No test for adler32 (adler32-zlib) [10045.929456] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [10046.023677] LNet: Added LNI 192.168.201.150@tcp [8/256/0/180] [10047.607167] Key type lgssc registered [10047.959245] Lustre: Echo OBD driver; http://www.lustre.org/ [10058.803324] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 10:56:39 (1788879399) [10061.090897] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10061.096461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10062.195420] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10062.205443] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10062.239508] Lustre: lustre-MDT0000: new disk, initializing [10062.261865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10062.268434] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10063.586957] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [10067.668631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10067.716064] Lustre: 326438:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10067.741310] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10067.743331] Lustre: Skipped 1 previous similar message [10067.778490] Lustre: lustre-MDT0001: new disk, initializing [10067.810788] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10067.822601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10067.826053] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10069.211815] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [10072.952370] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10074.897350] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10077.116602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10077.213671] Lustre: lustre-OST0000: new disk, initializing [10077.215317] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10077.218287] Lustre: 328405:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10077.246980] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10079.263309] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [10083.068671] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10084.343920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10084.347544] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10084.383805] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10094.562622] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10094.563282] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10094.566706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10094.570846] Lustre: Skipped 1 previous similar message [10099.683122] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10099.685811] Lustre: Skipped 2 previous similar messages [10099.736225] Lustre: server umount lustre-OST0000 complete [10103.775936] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10103.778353] LustreError: Skipped 1 previous similar message [10103.779667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10103.784195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10104.801807] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10107.157016] Lustre: server umount lustre-MDT0000 complete [10108.310181] LustreError: 326432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788879449 with bad export cookie 61657029228988621 [10108.313549] LustreError: 326432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10108.313886] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10108.417443] Lustre: server umount lustre-MDT0001 complete [10111.950875] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [10112.949264] Key type lgssc unregistered [10113.078444] LNet: 330313:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10113.080799] LNetError: 330313:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10113.088360] LNet: Removed LNI 192.168.201.150@tcp [10113.390513] Key type .llcrypt unregistered [10113.391650] Key type ._llcrypt unregistered [10121.291727] Key type ._llcrypt registered [10121.292891] Key type .llcrypt registered [10121.331154] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [10121.744831] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10121.772720] alg: No test for adler32 (adler32-zlib) [10122.620605] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [10122.706616] LNet: Added LNI 192.168.201.150@tcp [8/256/0/180] [10124.287196] Key type lgssc registered [10124.659957] Lustre: Echo OBD driver; http://www.lustre.org/ [10127.244316] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10127.249211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10128.374049] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10129.790533] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [10132.693979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10132.816488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10134.205046] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [10136.592896] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10138.417098] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10140.698270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10140.814819] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10142.779780] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [10145.244046] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10159.387801] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10159.924621] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10161.443748] Lustre: server umount lustre-MDT0000 complete [10162.726296] LustreError: 332292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788879503 with bad export cookie 4500264542489027095 [10162.730366] LustreError: 332292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10162.730579] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10164.704168] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10164.704601] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10164.708390] Lustre: Skipped 1 previous similar message [10164.711959] LustreError: 334047:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10164.715962] LustreError: 334047:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10166.124334] LustreError: 334047:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.50@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10166.129719] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.50@tcp (stopping) [10167.711859] LustreError: 331554:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10167.712181] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10168.877800] Lustre: server umount lustre-MDT0001 complete [10171.247694] Lustre: lustre-OST0000: Not available for connect from 192.168.201.50@tcp (stopping) [10176.340668] Lustre: server umount lustre-OST0000 complete [10179.161851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10180.481885] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [10181.440392] Lustre: 335755:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.150@tcp [10182.179410] Lustre: server umount MGS complete [10184.910065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10185.072954] LustreError: 336325:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10185.095171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10186.426686] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [10189.143973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10189.249739] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10190.527504] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [10193.417836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10195.580434] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [10198.206323] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10213.214962] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 14 sec [10215.078540] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10215.629841] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10218.827467] Lustre: server umount lustre-OST0000 complete [10220.512308] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10220.512898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10220.514888] Lustre: Skipped 1 previous similar message [10220.518268] Lustre: Skipped 1 previous similar message [10230.752187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10230.754391] Lustre: Skipped 3 previous similar messages [10230.777330] Lustre: server umount lustre-MDT0000 complete [10232.030257] LustreError: 336307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788879572 with bad export cookie 4500264542489027921 [10232.035045] LustreError: 336307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10232.035433] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10242.580159] Lustre: server umount lustre-MDT0001 complete [10246.461177] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [10247.508300] Key type lgssc unregistered [10247.647604] LNet: 339780:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10247.650531] LNetError: 339780:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10247.662318] LNet: Removed LNI 192.168.201.150@tcp [10247.992513] Key type .llcrypt unregistered [10247.993606] Key type ._llcrypt unregistered [10254.267525] Key type ._llcrypt registered [10254.269200] Key type .llcrypt registered [10254.306332] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_hostid [10260.750182] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [10261.058500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10261.090142] alg: No test for adler32 (adler32-zlib) [10261.951386] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [10262.048201] LNet: Added LNI 192.168.201.150@tcp [8/256/0/180] [10263.639196] Key type lgssc registered [10263.983879] Lustre: Echo OBD driver; http://www.lustre.org/ [10276.421722] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10276.426990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10277.508779] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10277.523074] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10277.561530] Lustre: lustre-MDT0000: new disk, initializing [10277.582652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10277.589449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10279.013509] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [10283.653562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10283.706776] Lustre: 342907:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10283.736331] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10283.738612] Lustre: Skipped 1 previous similar message [10283.780405] Lustre: lustre-MDT0001: new disk, initializing [10283.842417] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10283.857805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10283.861820] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10285.477978] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [10289.579395] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10291.704514] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10294.514538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10294.636902] Lustre: lustre-OST0000: new disk, initializing [10294.638798] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10294.640903] Lustre: 344876:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10294.672485] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10294.727046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10294.731117] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10294.776949] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10297.027348] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [10301.136561] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10302.647633] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10302.722427] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10304.129086] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10304.207101] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10304.992565] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10304.993761] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10304.995697] LustreError: Skipped 1 previous similar message [10304.996429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10305.000268] Lustre: Skipped 1 previous similar message [10310.115188] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10310.117392] Lustre: Skipped 2 previous similar messages [10315.234233] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10315.235996] Lustre: Skipped 1 previous similar message [10319.327126] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10319.384070] Lustre: server umount lustre-OST0000 complete [10324.960205] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10324.964038] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10324.969637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10326.747932] Lustre: server umount lustre-MDT0000 complete [10328.015228] LustreError: 342945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788879668 with bad export cookie 13752186830093031275 [10328.017848] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10328.018403] LustreError: 342945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10328.138339] Lustre: server umount lustre-MDT0001 complete [10331.185626] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 11:01:11 (1788879671) [10336.636394] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_hostid [10340.024849] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [10352.288685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10352.361918] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10352.371856] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10352.405732] Lustre: lustre-MDT0000: new disk, initializing [10352.426342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10352.432210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10353.788041] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [10357.902739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10357.942641] Lustre: 349637:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10357.966103] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10357.968157] Lustre: Skipped 1 previous similar message [10358.003808] Lustre: lustre-MDT0001: new disk, initializing [10358.036661] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10358.051427] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10358.055442] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10359.429372] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [10363.075191] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10364.900944] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10367.232490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10367.339205] Lustre: lustre-OST0000: new disk, initializing [10367.341399] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10367.344758] Lustre: 351606:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10367.372515] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10369.283329] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [10370.745349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10370.748893] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10370.778698] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10373.119704] Lustre: DEBUG MARKER: oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10386.402500] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10386.403047] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10386.406163] Lustre: Skipped 1 previous similar message [10386.409559] LustreError: Skipped 1 previous similar message [10386.411712] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10386.415240] Lustre: Skipped 3 previous similar messages [10388.745701] Lustre: server umount lustre-OST0000 complete [10391.520774] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10391.525245] Lustre: Skipped 2 previous similar messages [10396.567172] Lustre: server umount lustre-MDT0000 complete [10397.865277] LustreError: 349630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788879738 with bad export cookie 13752186830093032143 [10397.869018] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10397.869126] LustreError: 349630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10398.019715] Lustre: server umount lustre-MDT0001 complete [10401.769146] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [10402.817687] Key type lgssc unregistered [10402.941408] LNet: 353752:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10402.943652] LNetError: 353752:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10402.951265] LNet: Removed LNI 192.168.201.150@tcp [10403.256604] Key type .llcrypt unregistered [10403.257479] Key type ._llcrypt unregistered [10434.794035] Key type ._llcrypt registered [10434.795057] Key type .llcrypt registered [10434.831374] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [10434.933430] Key type .llcrypt unregistered [10434.934575] Key type ._llcrypt unregistered [10462.094117] Key type ._llcrypt registered [10462.095386] Key type .llcrypt registered [10462.128658] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [10462.228253] Key type .llcrypt unregistered [10462.229400] Key type ._llcrypt unregistered [10473.356320] Key type ._llcrypt registered [10473.357256] Key type .llcrypt registered [10473.394386] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [10473.492488] Key type .llcrypt unregistered [10473.494079] Key type ._llcrypt unregistered [10484.451046] Key type ._llcrypt registered [10484.452338] Key type .llcrypt registered [10484.486138] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [10484.585259] Key type .llcrypt unregistered [10484.586298] Key type ._llcrypt unregistered [10515.390938] Key type ._llcrypt registered [10515.392059] Key type .llcrypt registered [10515.432231] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [10515.534275] Key type .llcrypt unregistered [10515.535556] Key type ._llcrypt unregistered [10526.885031] Key type ._llcrypt registered [10526.886251] Key type .llcrypt registered [10526.923061] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [10527.028257] Key type .llcrypt unregistered [10527.029262] Key type ._llcrypt unregistered [10538.165844] Key type ._llcrypt registered [10538.167119] Key type .llcrypt registered [10538.204846] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [10538.302189] Key type .llcrypt unregistered [10538.303307] Key type ._llcrypt unregistered [10549.005218] Key type ._llcrypt registered [10549.006075] Key type .llcrypt registered [10549.039306] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [10549.127190] Key type .llcrypt unregistered [10549.128078] Key type ._llcrypt unregistered [10560.381359] Key type ._llcrypt registered [10560.382494] Key type .llcrypt registered [10560.419608] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [10560.518381] Key type .llcrypt unregistered [10560.519792] Key type ._llcrypt unregistered [10571.844116] Key type ._llcrypt registered [10571.845065] Key type .llcrypt registered [10571.879093] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [10571.976336] Key type .llcrypt unregistered [10571.977420] Key type ._llcrypt unregistered [10593.255082] Key type ._llcrypt registered [10593.256043] Key type .llcrypt registered [10593.290713] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [10593.378263] Key type .llcrypt unregistered [10593.379182] Key type ._llcrypt unregistered [10604.122394] Key type ._llcrypt registered [10604.123277] Key type .llcrypt registered [10604.166074] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [10604.278343] Key type .llcrypt unregistered [10604.279734] Key type ._llcrypt unregistered [10614.983691] Key type ._llcrypt registered [10614.984629] Key type .llcrypt registered [10615.014961] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [10615.107255] Key type .llcrypt unregistered [10615.108786] Key type ._llcrypt unregistered [10632.911061] Key type ._llcrypt registered [10632.911952] Key type .llcrypt registered [10632.946916] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_hostid [10638.982241] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [10639.321256] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10639.352730] alg: No test for adler32 (adler32-zlib) [10640.194030] Lustre: Lustre: Build Version: 2.17.58_39_g331ba7d [10640.273098] LNet: Added LNI 192.168.201.150@tcp [8/256/0/180] [10641.847196] Key type lgssc registered [10642.181080] Lustre: Echo OBD driver; http://www.lustre.org/ [10652.831571] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10652.835943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10653.926958] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [10653.935216] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [10653.968511] Lustre: tst250fs-MDT0000: new disk, initializing [10653.987467] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10653.994041] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10655.240696] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [10659.173061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10659.214270] Lustre: 373597:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10659.235905] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [10659.238246] Lustre: Skipped 1 previous similar message [10659.271767] Lustre: tst250fs-MDT0001: new disk, initializing [10659.301752] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10659.311890] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10659.315211] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10660.522679] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [10662.685739] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10664.745563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10664.845536] Lustre: tst250fs-OST0000: new disk, initializing [10664.847135] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [10664.849755] Lustre: 375225:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10664.883033] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10666.694234] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [10670.585759] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10670.638529] Lustre: tst250fs-OST0001: new disk, initializing [10670.639988] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [10670.642350] Lustre: 376090:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10670.669690] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10670.706170] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10670.712430] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10670.763608] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10670.775749] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10672.492838] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_default_debug -1 all [10675.033568] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [10686.434658] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10686.437101] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [10686.440116] Lustre: Skipped 2 previous similar messages [10686.442068] Lustre: Skipped 1 previous similar message [10691.357468] Lustre: server umount tst250fs-MDT0000 complete [10692.625305] LustreError: 373589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788880033 with bad export cookie 5539743402229745645 [10692.628879] LustreError: MGC192.168.201.150@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10692.629338] LustreError: 373589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10692.753417] Lustre: server umount tst250fs-MDT0001 complete [10704.903930] Lustre: server umount tst250fs-OST0000 complete [10716.388422] Lustre: server umount tst250fs-OST0001 complete [10722.263427] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_hostid [10724.782252] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [10734.625302] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 11:07:54 (1788880074) [10735.166179] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [10740.194576] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing set_hostid [10743.148222] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing load_modules_local [10753.248691] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 10537 sec ========= 11:08:13 (1788880093) [10753.794408] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 11:08:14 (1788880094) === [10754.857675] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 11:08:15 (1788880095) === [10767.281310] Lustre: DEBUG MARKER: oleg150-server.virtnet: executing unload_modules_local [10768.299800] Key type lgssc unregistered [10768.425315] LNet: 385202:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10768.427904] LNetError: 385202:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10768.434255] LNet: Removed LNI 192.168.201.150@tcp [10768.722108] Key type .llcrypt unregistered [10768.723087] Key type ._llcrypt unregistered