[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 600940767 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002394] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004012] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008012] pid_max: default: 32768 minimum: 301 [ 0.010045] LSM: Security Framework initializing [ 0.011052] Yama: becoming mindful. [ 0.013035] SELinux: Initializing. [ 0.014073] *** VALIDATE selinux *** [ 0.020455] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025284] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026139] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027119] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028105] *** VALIDATE tmpfs *** [ 0.030186] *** VALIDATE proc *** [ 0.031238] *** VALIDATE cgroup *** [ 0.032011] *** VALIDATE cgroup2 *** [ 0.034002] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035145] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037027] Spectre V2 : User space: Vulnerable [ 0.038011] Speculative Store Bypass: Vulnerable [ 0.040730] debug: unmapping init [mem 0xffffffffada59000-0xffffffffada60fff] [ 0.043000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043758] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044021] ... version: 2 [ 0.045012] ... bit width: 48 [ 0.046014] ... generic registers: 4 [ 0.047012] ... value mask: 0000ffffffffffff [ 0.048015] ... max period: 00007fffffffffff [ 0.049017] ... fixed-purpose events: 3 [ 0.050012] ... event mask: 000000070000000f [ 0.052216] rcu: Hierarchical SRCU implementation. [ 0.054453] smp: Bringing up secondary CPUs ... [ 0.055588] x86: Booting SMP configuration: [ 0.056028] .... node #0, CPUs: #1 #2 #3 [ 0.066167] smp: Brought up 1 node, 4 CPUs [ 0.068016] smpboot: Max logical packages: 1 [ 0.069015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.099024] node 0 deferred pages initialised in 28ms [ 0.107192] devtmpfs: initialized [ 0.109260] x86/mm: Memory block size: 128MB [ 0.113305] gcov: version magic: 0x41383552 [ 0.115225] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.118070] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.121363] pinctrl core: initialized pinctrl subsystem [ 0.124168] [ 0.124870] ************************************************************* [ 0.127011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.130013] ** ** [ 0.133014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.136023] ** ** [ 0.139014] ** This means that this kernel is built to expose internal ** [ 0.143014] ** IOMMU data structures, which may compromise security on ** [ 0.146010] ** your system. ** [ 0.148012] ** ** [ 0.151014] ** If you see this message and you are not debugging the ** [ 0.155014] ** kernel, report this immediately to your vendor! ** [ 0.158012] ** ** [ 0.161041] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.164013] ************************************************************* [ 0.169123] NET: Registered protocol family 16 [ 0.173166] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.180060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.185063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.193010] cpuidle: using governor menu [ 0.195645] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.196893] PCI: Using configuration type 1 for base access [ 0.197126] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.210175] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.216022] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.222057] cryptd: max_cpu_qlen set to 1000 [ 0.226243] ACPI: Added _OSI(Module Device) [ 0.229068] ACPI: Added _OSI(Processor Device) [ 0.231016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.234016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.242112] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.251455] ACPI: Interpreter enabled [ 0.253066] ACPI: PM: (supports S0 S3 S4 S5) [ 0.255013] ACPI: Using IOAPIC for interrupt routing [ 0.260115] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.267989] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.281133] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.284038] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.288018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.292074] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.297655] acpiphp: Slot [2] registered [ 0.299159] acpiphp: Slot [5] registered [ 0.301151] acpiphp: Slot [6] registered [ 0.302134] acpiphp: Slot [7] registered [ 0.304126] acpiphp: Slot [8] registered [ 0.306147] acpiphp: Slot [9] registered [ 0.308149] acpiphp: Slot [10] registered [ 0.310313] acpiphp: Slot [3] registered [ 0.312157] acpiphp: Slot [4] registered [ 0.314107] acpiphp: Slot [11] registered [ 0.315108] acpiphp: Slot [12] registered [ 0.317115] acpiphp: Slot [13] registered [ 0.319125] acpiphp: Slot [14] registered [ 0.321165] acpiphp: Slot [15] registered [ 0.323112] acpiphp: Slot [16] registered [ 0.324116] acpiphp: Slot [17] registered [ 0.326123] acpiphp: Slot [18] registered [ 0.328133] acpiphp: Slot [19] registered [ 0.330112] acpiphp: Slot [20] registered [ 0.332135] acpiphp: Slot [21] registered [ 0.334160] acpiphp: Slot [22] registered [ 0.336102] acpiphp: Slot [23] registered [ 0.337150] acpiphp: Slot [24] registered [ 0.339107] acpiphp: Slot [25] registered [ 0.341196] acpiphp: Slot [26] registered [ 0.342105] acpiphp: Slot [27] registered [ 0.344112] acpiphp: Slot [28] registered [ 0.346090] acpiphp: Slot [29] registered [ 0.348105] acpiphp: Slot [30] registered [ 0.349103] acpiphp: Slot [31] registered [ 0.351074] PCI host bridge to bus 0000:00 [ 0.353020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.356053] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.358024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.361024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.364022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.368026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.370194] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.374203] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.378548] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.388871] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.394050] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.397018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.400018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.403018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.408304] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.409907] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.413054] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.416989] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.424015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.440019] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.447011] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.453000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.463016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.478015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.502014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.525298] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.537017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.550015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.576016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.593022] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.603016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.611014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.630015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.649408] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.661014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.671016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.708016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.727000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.742014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.753033] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.778023] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.793685] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.806016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.821015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.849018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.866287] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.869540] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.873401] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.876446] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.879761] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.887097] iommu: Default domain type: Passthrough [ 0.888494] SCSI subsystem initialized [ 0.891172] ACPI: bus type USB registered [ 0.893097] usbcore: registered new interface driver usbfs [ 0.897084] usbcore: registered new interface driver hub [ 0.901079] usbcore: registered new device driver usb [ 0.903252] pps_core: LinuxPPS API ver. 1 registered [ 0.906011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.912064] PTP clock support registered [ 0.915102] EDAC MC: Ver: 3.0.0 [ 0.917197] PCI: Using ACPI for IRQ routing [ 0.918819] NetLabel: Initializing [ 0.921015] NetLabel: domain hash size = 128 [ 0.923051] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.926170] NetLabel: unlabeled traffic allowed by default [ 0.930003] vgaarb: loaded [ 0.932007] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.935014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.942000] clocksource: Switched to clocksource kvm-clock [ 1.075440] VFS: Disk quotas dquot_6.6.0 [ 1.078070] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.084077] *** VALIDATE ramfs *** [ 1.087821] *** VALIDATE hugetlbfs *** [ 1.091261] pnp: PnP ACPI init [ 1.093622] pnp: PnP ACPI: found 6 devices [ 1.119217] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.122810] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.125598] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.128658] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.132107] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.135262] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.138958] NET: Registered protocol family 2 [ 1.142752] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.148202] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.152220] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.165307] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.173195] TCP: Hash tables configured (established 65536 bind 65536) [ 1.179519] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.184844] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.190322] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.196456] NET: Registered protocol family 1 [ 1.201086] RPC: Registered named UNIX socket transport module. [ 1.204015] RPC: Registered udp transport module. [ 1.205721] RPC: Registered tcp transport module. [ 1.208626] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.212511] NET: Registered protocol family 44 [ 1.215758] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.218495] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.221891] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.224555] PCI: CLS 0 bytes, default 64 [ 1.226496] Unpacking initramfs... [ 2.822992] debug: unmapping init [mem 0xffff92107cc54000-0xffff92107ffbffff] [ 2.827197] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.829485] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.832597] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.416689] Initialise system trusted keyrings [ 3.419626] Key type blacklist registered [ 3.421328] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.430146] zbud: loaded [ 3.435625] *** VALIDATE nfs *** [ 3.438403] *** VALIDATE nfs4 *** [ 3.441575] pstore: using deflate compression [ 3.448654] Platform Keyring initialized [ 3.583273] NET: Registered protocol family 38 [ 3.585342] Key type asymmetric registered [ 3.587397] Asymmetric key parser 'x509' registered [ 3.590214] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.596303] io scheduler mq-deadline registered [ 3.598340] io scheduler kyber registered [ 3.600300] io scheduler bfq registered [ 3.602442] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.605976] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.610096] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.614805] ACPI: Power Button [PWRF] [ 3.623586] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.634424] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.655341] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.670738] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.701049] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.736971] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.771846] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.778579] Non-volatile memory driver v1.3 [ 3.780368] Linux agpgart interface v0.103 [ 3.825224] virtio_blk virtio1: [vda] 149944 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.829450] vda: detected capacity change from 0 to 76771328 [ 3.854147] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.858310] vdb: detected capacity change from 0 to 1073741824 [ 3.881432] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.887783] vdc: detected capacity change from 0 to 2621440000 [ 3.909530] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.913702] vdd: detected capacity change from 0 to 2621440000 [ 3.938344] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.943167] vde: detected capacity change from 0 to 4294967296 [ 3.962283] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.965727] vdf: detected capacity change from 0 to 4294967296 [ 3.974784] libphy: Fixed MDIO Bus: probed [ 3.981409] usbcore: registered new interface driver usbserial_generic [ 3.983828] usbserial: USB Serial support registered for generic [ 3.986855] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.996321] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.000157] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.005260] mousedev: PS/2 mouse device common for all mice [ 4.009532] rtc_cmos 00:05: RTC can wake from S4 [ 4.013078] rtc_cmos 00:05: registered as rtc0 [ 4.015168] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.018389] intel_pstate: CPU model not supported [ 4.022039] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.025827] hid: raw HID events driver (C) Jiri Kosina [ 4.028247] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.031535] usbcore: registered new interface driver usbhid [ 4.031542] usbhid: USB HID core driver [ 4.031644] drop_monitor: Initializing network drop monitor service [ 4.031773] Initializing XFRM netlink socket [ 4.033239] NET: Registered protocol family 10 [ 4.036391] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.051515] Segment Routing with IPv6 [ 4.053079] NET: Registered protocol family 17 [ 4.054971] mpls_gso: MPLS GSO support [ 4.060969] RAS: Correctable Errors collector initialized. [ 4.064929] AVX version of gcm_enc/dec engaged. [ 4.068627] AES CTR mode by8 optimization enabled [ 4.166797] sched_clock: Marking stable (4166778750, 0)->(5203594396, -1036815646) [ 4.171788] registered taskstats version 1 [ 4.176459] Loading compiled-in X.509 certificates [ 4.178616] zswap: loaded using pool lzo/zbud [ 4.215858] Key type big_key registered [ 4.232969] Key type encrypted registered [ 4.235437] ima: No TPM chip found, activating TPM-bypass! [ 4.238842] ima: Allocated hash algorithm: sha1 [ 4.241569] ima: No architecture policies found [ 4.244339] evm: Initialising EVM extended attributes: [ 4.247302] evm: security.selinux [ 4.249139] evm: security.ima [ 4.250880] evm: security.capability [ 4.252651] evm: HMAC attrs: 0x1 [ 4.256509] rtc_cmos 00:05: setting system clock to 2026-09-16 22:01:00 UTC (1789596060) [ 4.266567] debug: unmapping init [mem 0xffffffffaea03000-0xffffffffaebfffff] [ 4.270261] debug: unmapping init [mem 0xffffffffad782000-0xffffffffada58fff] [ 4.282400] Write protecting the kernel read-only data: 28672k [ 4.287352] debug: unmapping init [mem 0xffffffffabe03000-0xffffffffabffffff] [ 4.291732] debug: unmapping init [mem 0xffffffffac714000-0xffffffffac7fffff] [ 4.343697] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.356372] systemd[1]: Detected virtualization kvm. [ 4.358793] systemd[1]: Detected architecture x86-64. [ 4.361495] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.391401] systemd[1]: No hostname configured. [ 4.393621] systemd[1]: Set hostname to . [ 4.397482] random: systemd: uninitialized urandom read (16 bytes read) [ 4.399975] systemd[1]: Initializing machine ID from random generator. [ 4.477357] random: ln: uninitialized urandom read (6 bytes read) [ 4.611830] random: systemd: uninitialized urandom read (16 bytes read) [ 4.614864] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 4.620857] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 4.627588] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.657806] device-mapper: uevent: version 1.0.3 [ 5.660679] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] [ 6.554465] virtio_net virtio0 ens2: renamed from eth0 Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 6.615979] random: fast init done [ 6.737089] scsi host0: ata_piix [ 6.756320] scsi host1: ata_piix [ 6.841960] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.849719] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 11.764854] random: crng init done [ 11.768105] random: 7 urandom warning(s) missed due to ratelimiting [ 13.123585] dracut-initqueue[591]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 13.962508] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 15.515340] printk: systemd: 26 output lines suppressed due to ratelimiting [ 15.854432] SELinux: Disabled at runtime. [ 15.920825] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 15.932664] systemd[1]: Detected virtualization kvm. [ 15.938375] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 16.689130] systemd[1]: initrd-switch-root.service: Succeeded. [ 16.692982] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 16.709187] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 16.714549] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 16.718797] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 16.729945] systemd[1]: Starting Journal Service... Starting Journal Service... [ 16.740129] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Huge Pages File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Paths. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. [ OK ] Stopped target Initrd File Systems. Starting Apply Kernel Variables... Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Local Encrypted Volumes. [ 16.891820] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting POSIX Message Queue File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting Kernel Debug File System... Starting udev Coldplug all Devices... [ OK ] Listening on Process Core Dump Socket. Starting Remount Root and Kernel File Systems... [ OK ] Mounted Huge Pages File System. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 17.701678] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 18.444780] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 18.532360] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 19.032278] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 19.138400] EDAC sbridge: Ver: 1.1.2 [ 21.560703] Key type dns_resolver registered [ 21.968533] NFS: Registering the id_resolver key type [ 21.972504] Key type id_resolver registered [ 21.976751] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg655-server login: [ 74.629126] hrtimer: interrupt took 2883412 ns [ 83.696717] libcfs: loading out-of-tree module taints kernel. [ 83.798505] Key type ._llcrypt registered [ 83.807792] Key type .llcrypt registered [ 84.073979] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 107.040604] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 108.716260] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 108.728962] alg: No test for adler32 (adler32-zlib) [ 110.203940] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [ 110.862749] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 112.727265] Key type lgssc registered [ 114.352114] Lustre: Echo OBD driver; http://www.lustre.org/ [ 131.519545] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 171.238992] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 184.286980] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 184.304970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 185.628276] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 185.674705] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 185.813204] Lustre: lustre-MDT0000: new disk, initializing [ 185.904492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 185.933228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 190.122817] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 202.537543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 202.645515] Lustre: 6529:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 202.690928] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 202.695285] Lustre: Skipped 1 previous similar message [ 202.756604] Lustre: lustre-MDT0001: new disk, initializing [ 202.837213] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 202.877856] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 202.888509] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 207.156535] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 212.003979] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 220.816141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 221.061651] Lustre: lustre-OST0000: new disk, initializing [ 221.065931] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 221.071805] Lustre: 8467:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 221.137044] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 225.810145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 225.824584] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 225.930038] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 226.443328] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 240.640744] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 240.789483] Lustre: lustre-OST0001: new disk, initializing [ 240.792561] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 240.797107] Lustre: 9540:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 240.841343] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 247.296814] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 248.380055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 248.402524] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 248.428858] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 259.744980] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 267.499819] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 274.815422] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing check_logdir /tmp/testlogs/ [ 280.900514] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing yml_node [ 285.349409] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 288.066353] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 290.991604] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 293.119659] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Sep 16 18:05:48 EDT 2026 [ 312.082405] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 313.647670] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 330.215215] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 330.216961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 330.232974] Lustre: Skipped 2 previous similar messages [ 330.240687] Lustre: Skipped 1 previous similar message [ 334.947644] Lustre: server umount lustre-MDT0000 complete [ 338.532994] LustreError: 6521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789596394 with bad export cookie 4574702324277411218 [ 338.533842] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 338.549466] LustreError: 6521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 340.447991] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 340.450504] LustreError: 8475:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 340.450686] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 340.450690] Lustre: Skipped 1 previous similar message [ 340.473886] Lustre: Skipped 2 previous similar messages [ 340.517915] LustreError: 8475:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 343.455955] LustreError: 6538:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 343.456505] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 343.480908] Lustre: Skipped 2 previous similar messages [ 345.093328] Lustre: server umount lustre-MDT0001 complete [ 349.332745] Lustre: server umount lustre-OST0000 complete [ 353.244904] Lustre: server umount lustre-OST0001 complete [ 360.399802] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 371.622510] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 412.323984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 412.643337] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 412.695692] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 412.773761] Lustre: lustre-MDT0000: new disk, initializing [ 412.869643] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 412.889750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 417.935302] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 430.238941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 430.379933] Lustre: 15097:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 430.461607] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 430.466174] Lustre: Skipped 1 previous similar message [ 430.617462] Lustre: lustre-MDT0001: new disk, initializing [ 430.757492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 430.808790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 430.828069] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 435.582217] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 446.093253] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 453.570956] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 461.854939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 462.192763] Lustre: lustre-OST0000: new disk, initializing [ 462.196107] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 462.207820] Lustre: 17067:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 462.351396] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 463.809817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 463.825675] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 463.942996] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 468.933197] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 479.459108] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 484.700797] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 484.991310] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 490.385837] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 490.614413] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 494.561686] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 494.571849] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 494.572094] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 494.576711] Lustre: Skipped 1 previous similar message [ 494.608334] Lustre: Skipped 1 previous similar message [ 499.702325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 499.712039] Lustre: Skipped 1 previous similar message [ 506.847688] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 507.110576] Lustre: server umount lustre-OST0000 complete [ 512.992351] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 513.002057] LustreError: Skipped 1 previous similar message [ 513.010408] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 513.037087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 513.041849] Lustre: Skipped 2 previous similar messages [ 517.281894] Lustre: server umount lustre-MDT0000 complete [ 520.161800] LustreError: 16425:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 520.186561] LustreError: 16425:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 521.289627] LustreError: 15087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789596577 with bad export cookie 4574702324277412751 [ 521.290550] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 521.316098] LustreError: 15087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 521.675886] Lustre: server umount lustre-MDT0001 complete [ 527.928878] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 18:09:43 (1789596583) [ 550.351645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 551.058784] LustreError: 19943:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 551.086428] LustreError: 19943:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 551.165641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 556.268181] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 556.513757] LustreError: 19944:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 565.674569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 570.937595] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 579.181915] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 585.329886] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 587.679798] Lustre: Failing over lustre-MDT0000 [ 588.057632] Lustre: server umount lustre-MDT0000 complete [ 591.779677] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 591.796525] Lustre: Skipped 1 previous similar message [ 591.800993] LustreError: 20671:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 591.829280] LustreError: 20671:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 608.223168] Lustre: 3663:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789596648/real 1789596648] req@ffff9210f8d66300 x1876527590570624/t0(0) o400->MGC192.168.206.155@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789596664 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 608.254775] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 608.296370] LustreError: 19943:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 608.325648] LustreError: 19943:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 609.007477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 619.489029] LustreError: 3660:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9210fef91f80 x1876527590573824/t0(0) o250->MGC192.168.206.155@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 619.893334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 619.898960] Lustre: Skipped 1 previous similar message [ 619.953653] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 625.126845] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 625.131879] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 625.164649] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 625.511920] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 636.176919] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 642.529282] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 642.539965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 642.552454] Lustre: Skipped 2 previous similar messages [ 642.573493] Lustre: Skipped 3 previous similar messages [ 646.906622] Lustre: server umount lustre-MDT0000 complete [ 647.672044] LustreError: 19940:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 647.698363] LustreError: 19940:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 651.600176] LustreError: 19923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789596707 with bad export cookie 4574702324277414165 [ 651.618780] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 651.624030] LustreError: 19923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 652.354194] Lustre: server umount lustre-MDT0001 complete [ 663.234495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 669.047653] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 678.731489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 684.435474] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 692.563664] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 699.078868] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 701.766466] Lustre: Failing over lustre-MDT0000 [ 702.143554] Lustre: server umount lustre-MDT0000 complete [ 703.860419] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 703.899630] Lustre: Skipped 1 previous similar message [ 714.211464] LustreError: 23278:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 714.242766] LustreError: 23278:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 720.353035] Lustre: 3664:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789596760/real 1789596760] req@ffff9210f8df6300 x1876527590601216/t0(0) o400->MGC192.168.206.155@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789596776 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 720.412882] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 723.469127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 730.725226] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x3f7c9b86afb58cbf [ 730.755055] Lustre: MGC192.168.206.155@tcp: Connection restored to 0@lo (at 0@lo) [ 730.758234] Lustre: Skipped 1 previous similar message [ 731.163795] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 736.232105] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 736.237147] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 736.285461] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 737.102807] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 747.642377] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 754.659963] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 754.665490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 754.685483] Lustre: Skipped 1 previous similar message [ 754.700575] Lustre: Skipped 1 previous similar message [ 758.349894] Lustre: server umount lustre-MDT0000 complete [ 762.849402] LustreError: 23261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789596819 with bad export cookie 4574702324277415103 [ 762.855649] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 762.874274] LustreError: 23261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 763.434728] Lustre: server umount lustre-MDT0001 complete [ 773.970315] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 18:13:49 (1789596829) [ 792.515156] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 802.295819] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 851.808490] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 863.581957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 863.782541] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 863.806329] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 863.873794] Lustre: lustre-MDT0000: new disk, initializing [ 863.931185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 863.937130] Lustre: Skipped 3 previous similar messages [ 863.952807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 868.823049] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 879.765378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 879.901251] Lustre: 29662:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 880.129624] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 884.703531] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 894.003977] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 899.771293] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 906.794362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 907.149610] Lustre: 31632:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 908.985792] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 909.038625] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 909.091618] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 909.125406] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 913.495553] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 923.281945] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 932.181470] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 955.361815] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 955.367266] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 955.409378] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 955.429294] Lustre: Skipped 1 previous similar message [ 957.565566] Lustre: server umount lustre-OST0000 complete [ 962.019753] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 976.351428] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 976.576788] Lustre: server umount lustre-MDT0000 complete [ 979.945653] LustreError: 29672:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 979.972330] LustreError: 29672:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 13 previous similar messages [ 980.167728] LustreError: 29654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789597036 with bad export cookie 4574702324277415516 [ 980.174093] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 980.186160] LustreError: 29654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 980.980939] Lustre: server umount lustre-MDT0001 complete [ 993.473265] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 996.124418] Key type lgssc unregistered [ 996.444748] LNet: 33608:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 996.454182] LNetError: 33608:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 996.476118] LNet: Removed LNI 192.168.206.155@tcp [ 997.326133] Key type .llcrypt unregistered [ 997.327956] Key type ._llcrypt unregistered [ 1024.691792] Key type ._llcrypt registered [ 1024.698485] Key type .llcrypt registered [ 1024.864399] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 1026.328865] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1026.382367] alg: No test for adler32 (adler32-zlib) [ 1027.566484] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [ 1027.864881] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 1029.520214] Key type lgssc registered [ 1030.548451] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1038.528767] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1038.563699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1040.129192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1044.766082] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1050.604746] LustreError: 35021:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1050.624420] LustreError: 35021:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1054.333443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1054.802840] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1059.740090] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1068.169979] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1074.733954] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1082.323448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1082.879676] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1089.214430] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1098.166233] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1099.283213] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 1115.447202] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 18:19:30 (1789597170) [ 1117.561262] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1119.103963] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1125.507543] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 18:19:41 (1789597181) [ 1134.408740] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 18:19:49 (1789597189) [ 1135.601240] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1144.071544] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1144.079182] Lustre: Skipped 2 previous similar messages [ 1146.073586] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 18:20:01 (1789597201) [ 1149.893749] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1149.899708] Lustre: Skipped 3 previous similar messages [ 1168.978332] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1168.989826] Lustre: Skipped 6 previous similar messages [ 1171.989827] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 18:20:27 (1789597227) [ 1175.110666] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1178.962696] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1188.385921] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 18:20:43 (1789597243) [ 1190.536599] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1190.546108] Lustre: Skipped 3 previous similar messages [ 1204.165911] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 18:20:59 (1789597259) [ 1222.114298] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1222.140446] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1222.141234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1222.160151] Lustre: Skipped 1 previous similar message [ 1227.234775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1227.246488] Lustre: Skipped 1 previous similar message [ 1232.354427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1232.363209] Lustre: Skipped 1 previous similar message [ 1233.375463] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1233.582384] Lustre: server umount lustre-OST0000 complete [ 1237.474973] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1237.487354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1237.494430] Lustre: Skipped 2 previous similar messages [ 1237.525362] Lustre: Skipped 1 previous similar message [ 1242.600949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1242.608429] Lustre: Skipped 1 previous similar message [ 1243.520092] Lustre: server umount lustre-MDT0000 complete [ 1247.375513] LustreError: 35004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789597303 with bad export cookie 13547545093151159863 [ 1247.386992] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1247.399521] LustreError: 35004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1247.784393] Lustre: server umount lustre-MDT0001 complete [ 1259.180325] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 1261.664754] Key type lgssc unregistered [ 1261.935693] LNet: 42866:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1261.958092] LNetError: 42866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1261.973171] LNet: Removed LNI 192.168.206.155@tcp [ 1262.988146] Key type .llcrypt unregistered [ 1262.990427] Key type ._llcrypt unregistered [ 1279.537200] Key type ._llcrypt registered [ 1279.543838] Key type .llcrypt registered [ 1279.657157] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 1293.937753] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 1294.666204] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1294.697488] alg: No test for adler32 (adler32-zlib) [ 1295.736758] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [ 1295.956783] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 1297.624497] Key type lgssc registered [ 1298.533775] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1334.590938] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1334.616488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1335.897708] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1335.922732] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1336.004829] Lustre: lustre-MDT0000: new disk, initializing [ 1336.085358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1336.125508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1340.307203] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1350.369155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1350.556612] Lustre: 46017:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1350.639465] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1350.642790] Lustre: Skipped 1 previous similar message [ 1350.693748] Lustre: lustre-MDT0001: new disk, initializing [ 1350.755455] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1350.788051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1350.795426] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1355.024542] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1364.727120] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1370.773524] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1378.276562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1378.527603] Lustre: lustre-OST0000: new disk, initializing [ 1378.537433] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1378.547310] Lustre: 47984:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1378.678111] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1385.285760] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1388.594188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1388.602383] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1388.724940] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1395.248783] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1399.282241] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1399.502849] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1403.778071] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1403.955431] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1409.504687] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1409.505373] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1409.520133] LustreError: Skipped 1 previous similar message [ 1409.539558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1409.545648] Lustre: Skipped 1 previous similar message [ 1414.642363] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1414.652759] Lustre: Skipped 1 previous similar message [ 1419.748387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1419.756117] Lustre: Skipped 1 previous similar message [ 1420.255135] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1420.492488] Lustre: server umount lustre-OST0000 complete [ 1424.866350] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1424.880218] Lustre: Skipped 1 previous similar message [ 1424.883351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1429.987351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1429.993357] Lustre: Skipped 2 previous similar messages [ 1430.273382] Lustre: server umount lustre-MDT0000 complete [ 1433.809562] LustreError: 46009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789597490 with bad export cookie 5236351578548790239 [ 1433.812685] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1433.820604] LustreError: 46009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1434.271591] Lustre: server umount lustre-MDT0001 complete [ 1444.007201] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 18:24:59 (1789597499) [ 1450.920676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1451.614985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1456.163338] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1456.610123] LustreError: 50446:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1456.636092] LustreError: 50446:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1461.731744] LustreError: 50447:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1464.744806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1465.116933] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1469.320867] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1476.033346] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1481.298853] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1488.420667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1488.862398] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1494.863676] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1502.992512] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1507.574130] LustreError: 52974:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1508.541610] Lustre: Modifying parameter general.timeout=1 in log params [ 1510.169794] Lustre: Modifying parameter general.timeout=3 in log params [ 1510.177346] Lustre: Skipped 1 previous similar message [ 1512.540560] Lustre: Modifying parameter general.timeout=6 in log params [ 1512.547921] Lustre: Skipped 2 previous similar messages [ 1516.653665] Lustre: Modifying parameter general.timeout=11 in log params [ 1516.659774] Lustre: Skipped 4 previous similar messages [ 1525.087286] Lustre: Modifying parameter general.timeout=22 in log params [ 1525.091589] Lustre: Skipped 10 previous similar messages [ 1541.465710] Lustre: Modifying parameter general.timeout=42 in log params [ 1541.472344] Lustre: Skipped 19 previous similar messages [ 1574.039387] Lustre: Modifying parameter general.timeout=83 in log params [ 1574.043810] Lustre: Skipped 40 previous similar messages [ 1619.275492] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 18:27:55 (1789597675) [ 1621.513187] Lustre: 59908:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1623.060656] Lustre: 60007:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1626.320491] Lustre: 60206:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1626.335444] Lustre: 60206:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1639.160772] Lustre: Disabling parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 in log params [ 1639.172092] Lustre: Skipped 61 previous similar messages [ 1653.385779] Lustre: 61774:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1653.390782] Lustre: 61774:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1653.405515] LustreError: 61774:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1657.669735] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 18:28:33 (1789597713) [ 1670.118424] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1670.127066] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1670.145140] Lustre: Skipped 1 previous similar message [ 1670.149594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1675.960840] Lustre: server umount lustre-MDT0000 complete [ 1679.397441] LustreError: 53023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789597735 with bad export cookie 5236351578548791128 [ 1679.412471] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1679.414618] LustreError: 53023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1679.636769] Lustre: server umount lustre-MDT0001 complete [ 1693.854210] Lustre: server umount lustre-OST0000 complete [ 1723.122125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1723.399933] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1723.842931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1729.126258] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1738.842573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1738.953980] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1738.983188] Lustre: 64308:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1743.434527] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1749.984500] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1754.732325] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1761.139222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1761.345077] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1761.566227] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1761.576239] Lustre: Skipped 1 previous similar message [ 1766.925862] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1775.125455] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1776.946352] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1776.964636] Lustre: Skipped 8 previous similar messages [ 1777.033713] Lustre: 66872:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1786.851377] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1786.870689] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1786.893015] Lustre: Skipped 2 previous similar messages [ 1786.908443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1786.914926] Lustre: Skipped 5 previous similar messages [ 1789.550964] Lustre: server umount lustre-OST0000 complete [ 1795.551962] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1795.559590] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1795.571629] Lustre: Skipped 1 previous similar message [ 1807.327111] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1807.589870] Lustre: server umount lustre-MDT0000 complete [ 1807.849624] LustreError: 64318:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1811.749283] LustreError: 67331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789597867 with bad export cookie 5236351578548794012 [ 1811.756526] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1811.762056] LustreError: 67331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1812.095887] Lustre: server umount lustre-MDT0001 complete [ 1824.116694] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 1826.537376] Key type lgssc unregistered [ 1826.799598] LNet: 68083:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1826.808500] LNetError: 68083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1826.824396] LNet: Removed LNI 192.168.206.155@tcp [ 1827.608802] Key type .llcrypt unregistered [ 1827.610960] Key type ._llcrypt unregistered [ 1852.688642] Key type ._llcrypt registered [ 1852.690509] Key type .llcrypt registered [ 1852.854662] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 1854.283470] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1854.363986] alg: No test for adler32 (adler32-zlib) [ 1855.476914] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [ 1855.760521] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 1857.473218] Key type lgssc registered [ 1858.478331] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1866.655994] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1866.676809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1868.155201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1872.403592] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1878.501979] LustreError: 69513:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1878.520626] LustreError: 69513:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1880.205922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1880.773890] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1884.870845] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1892.686817] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1898.070907] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1904.696982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1905.281635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1911.598619] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1919.301377] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1926.418628] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1936.867806] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1936.877253] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1936.902464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1939.424469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1939.426023] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1939.457864] Lustre: Skipped 1 previous similar message [ 1941.029906] Lustre: server umount lustre-MDT0000 complete [ 1944.536711] LustreError: 72194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789598000 with bad export cookie 7655201415267585053 [ 1944.546489] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1944.558058] LustreError: 72194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1944.558254] LustreError: 69507:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1944.566319] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1944.621335] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1944.626477] Lustre: Skipped 2 previous similar messages [ 1944.839893] Lustre: server umount lustre-MDT0001 complete [ 1950.895191] Lustre: server umount lustre-OST0000 complete [ 1979.256826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1979.463716] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1979.483252] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1979.683928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1984.039445] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1992.477180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1992.688773] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1992.781306] Lustre: 74382:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1993.256367] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1998.138032] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2005.832179] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2011.941403] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2019.997899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2020.336026] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2020.894605] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2027.910860] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2036.258511] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2038.072426] Lustre: 76954:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2040.457900] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2047.458371] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2047.464079] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2047.480151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2051.574109] Lustre: server umount lustre-OST0000 complete [ 2058.209261] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2058.213995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2058.220978] Lustre: Skipped 2 previous similar messages [ 2058.237844] Lustre: Skipped 2 previous similar messages [ 2061.965512] Lustre: server umount lustre-MDT0000 complete [ 2063.328737] LustreError: 74392:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2063.344281] LustreError: 74392:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2065.622783] LustreError: 74373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789598121 with bad export cookie 7655201415267585900 [ 2065.633914] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2065.639669] LustreError: 74373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2065.978516] Lustre: server umount lustre-MDT0001 complete [ 2078.282283] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 2080.921940] Key type lgssc unregistered [ 2081.290770] LNet: 78167:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2081.302368] LNetError: 78167:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2081.319833] LNet: Removed LNI 192.168.206.155@tcp [ 2082.290561] Key type .llcrypt unregistered [ 2082.293192] Key type ._llcrypt unregistered [ 2108.376677] Key type ._llcrypt registered [ 2108.381324] Key type .llcrypt registered [ 2108.535408] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 2109.597684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2109.632337] alg: No test for adler32 (adler32-zlib) [ 2110.637189] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [ 2110.844111] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 2112.535321] Key type lgssc registered [ 2113.732341] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2120.983987] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2120.997861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2122.694096] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2127.026922] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2131.943285] LustreError: 79578:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2131.975844] LustreError: 79578:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2136.113937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2136.713575] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2141.145544] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2148.822598] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2155.255801] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2163.180963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2163.876162] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2169.850860] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2177.343334] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2187.372303] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2188.476378] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2189.787790] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2192.438754] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2192.450348] Lustre: Skipped 1 previous similar message [ 2196.548309] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2196.553461] Lustre: Skipped 2 previous similar messages [ 2204.786358] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2204.792875] Lustre: Skipped 6 previous similar messages [ 2220.836359] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2220.842651] Lustre: Skipped 14 previous similar messages [ 2253.045935] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2253.052800] Lustre: Skipped 33 previous similar messages [ 2296.115478] Lustre: 87689:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2298.715808] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 18:39:14 (1789598354) [ 2300.466920] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2302.085509] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 18:39:17 (1789598357) [ 2305.509049] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2305.515947] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2305.537904] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2309.601820] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2309.605861] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2309.609508] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2309.626201] Lustre: Skipped 1 previous similar message [ 2311.276696] Lustre: server umount lustre-OST0000 complete [ 2314.720899] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2314.735629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2314.741298] Lustre: Skipped 1 previous similar message [ 2319.848391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2319.857224] Lustre: Skipped 1 previous similar message [ 2320.713707] Lustre: server umount lustre-MDT0000 complete [ 2324.496431] LustreError: 81432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789598380 with bad export cookie 263484490296758703 [ 2324.502497] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2324.507955] LustreError: 81432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2324.789506] Lustre: server umount lustre-MDT0001 complete [ 2335.214989] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 2338.047129] Key type lgssc unregistered [ 2338.348726] LNet: 88936:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2338.357596] LNetError: 88936:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2338.370840] LNet: Removed LNI 192.168.206.155@tcp [ 2339.326288] Key type .llcrypt unregistered [ 2339.329466] Key type ._llcrypt unregistered [ 2343.413776] Key type ._llcrypt registered [ 2343.416615] Key type .llcrypt registered [ 2343.521878] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2347.568128] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules [ 2348.225506] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2348.263124] alg: No test for adler32 (adler32-zlib) [ 2349.288135] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [ 2349.474092] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 2351.127635] Key type lgssc registered [ 2352.267775] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2353.703995] LustreError: 90133:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2371.616211] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 2375.810178] LustreError: 90133:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2381.049641] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2381.080449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2382.727712] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2386.913540] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2392.032431] LustreError: 91159:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2392.056323] LustreError: 91159:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2394.499291] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 18:40:50 (1789598450) [ 2397.153293] LustreError: 91158:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2398.958319] LustreError: 91190:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2398.974184] LustreError: 91190:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2398.986368] LustreError: 91190:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 0, failed: rc = -5 [ 2399.350519] Lustre: server umount lustre-MDT0000 complete [ 2412.917740] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 2415.594730] Key type lgssc unregistered [ 2415.871554] LNet: 92682:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2415.877384] LNetError: 92682:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2416.938176] LNet: Removed LNI 192.168.206.155@tcp [ 2417.712500] Key type .llcrypt unregistered [ 2417.716587] Key type ._llcrypt unregistered [ 2433.800669] Key type ._llcrypt registered [ 2433.803284] Key type .llcrypt registered [ 2433.920510] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 2434.886589] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2435.025964] alg: No test for adler32 (adler32-zlib) [ 2436.110833] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [ 2436.286075] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 2437.927359] Key type lgssc registered [ 2438.823786] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2446.297152] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2446.311391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2447.830853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2452.171524] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2458.081417] LustreError: 93895:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2458.105291] LustreError: 93895:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2461.640939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2462.397519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2466.871720] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2475.834299] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2482.135611] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2489.859574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2490.741608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2497.810712] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2507.458849] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2559.635513] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 18:43:35 (1789598615) [ 2561.449645] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2563.607228] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 18:43:38 (1789598618) [ 2568.674487] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2568.677565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2568.698036] Lustre: Skipped 2 previous similar messages [ 2568.716854] Lustre: Skipped 2 previous similar messages [ 2573.796337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2573.805423] Lustre: Skipped 2 previous similar messages [ 2574.595835] Lustre: server umount lustre-MDT0000 complete [ 2578.449732] LustreError: 93877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789598634 with bad export cookie 12727250792792096655 [ 2578.463317] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2578.473511] LustreError: 93877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2578.814619] Lustre: server umount lustre-MDT0001 complete [ 2592.874753] Lustre: server umount lustre-OST0000 complete [ 2603.582142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2604.021868] LustreError: 98198:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2604.231823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2608.543251] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2609.632927] LustreError: 98199:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2614.754118] LustreError: 98198:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2616.370041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2617.054501] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2621.876348] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2629.733231] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2634.852498] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2647.144991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2647.308685] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2647.316444] LustreError: 98187:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2647.335906] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2647.346835] LustreError: 100281:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2647.353100] LustreError: 100281:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2647.358609] LustreError: 100281:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2647.408347] Lustre: server umount lustre-OST0000 complete [ 2647.411854] LustreError: 100281:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2655.728959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2655.958609] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2655.967249] LustreError: 98185:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2655.982477] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2655.991268] LustreError: 100742:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2656.000247] LustreError: 100742:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2656.008073] LustreError: 100742:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2656.093973] Lustre: server umount lustre-OST0000 complete [ 2656.097298] LustreError: 100742:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2666.225360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2666.417288] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2666.421497] Lustre: Found index 0 for lustre-OST0000, updating log [ 2666.427929] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2666.490822] Lustre: lustre-OST0000: new disk, initializing [ 2666.494495] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2666.499518] Lustre: 101276:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2666.830250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2674.205944] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2678.373655] Lustre: Failing over lustre-OST0000 [ 2678.501415] Lustre: server umount lustre-OST0000 complete [ 2683.363206] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2683.380367] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2683.405805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2684.386837] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2686.434331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2686.441795] Lustre: Skipped 3 previous similar messages [ 2691.552823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2691.562877] Lustre: Skipped 1 previous similar message [ 2692.680170] Lustre: server umount lustre-MDT0000 complete [ 2696.247331] LustreError: 98178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789598752 with bad export cookie 12727250792792099399 [ 2696.248198] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2696.253170] LustreError: 98178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 2696.452039] Lustre: server umount lustre-MDT0001 complete [ 2706.076716] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 18:46:01 (1789598761) [ 2723.713540] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 2733.874822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2734.140627] LustreError: 103619:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2734.218216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2738.492736] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2743.782208] LustreError: 103620:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2743.807595] LustreError: 103620:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2747.874193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2753.478474] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2756.446674] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2762.639710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2763.546063] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2768.960394] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2772.799320] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2772.809267] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2772.845118] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2777.321592] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2777.394803] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2777.486664] Lustre: lustre-OST0001: new disk, initializing [ 2777.491636] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2777.499229] Lustre: 105937:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2777.650801] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 2777.675503] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2777.817421] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2777.822595] Lustre: Skipped 2 previous similar messages [ 2778.865823] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:03 [ 2778.880526] Lustre: lustre-OST0000: Recovery over after 0:06, of 1 clients 1 recovered and 0 were evicted. [ 2778.880734] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2778.889770] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2778.894286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2778.918919] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2778.959070] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2784.406225] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2792.790715] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2803.963914] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2809.826898] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2809.837674] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2809.863775] Lustre: Skipped 2 previous similar messages [ 2809.884840] Lustre: Skipped 2 previous similar messages [ 2814.320320] Lustre: server umount lustre-MDT0001 complete [ 2814.952232] LustreError: 103620:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2814.967185] LustreError: 103620:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2822.953526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2823.027335] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2823.031619] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2823.105257] Lustre: 103607:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2824.558245] LustreError: 107506:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2824.571054] LustreError: 107506:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2824.584731] LustreError: 103611:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2824.751834] LustreError: 107519:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.155@tcp: cfg command failed: rc = -17 [ 2824.768228] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2824.768228] [ 2828.212620] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2828.808474] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2838.045793] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 18:48:13 (1789598893) [ 2858.115940] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 2877.564209] Lustre: 109556:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2899.082157] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2902.875967] Lustre: 110691:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2992.608868] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2992.613146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2992.629020] Lustre: Skipped 1 previous similar message [ 2992.652669] Lustre: Skipped 4 previous similar messages [ 2995.698713] Lustre: server umount lustre-MDT0000 complete [ 2997.737756] LustreError: 104838:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2997.755941] LustreError: 104838:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 2999.975994] LustreError: 103600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789599056 with bad export cookie 12727250792792100330 [ 3000.000268] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3000.012271] LustreError: 103600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3000.342319] Lustre: server umount lustre-MDT0001 complete [ 3013.777286] Lustre: server umount lustre-OST0000 complete [ 3028.608808] Lustre: server umount lustre-OST0001 complete [ 3034.939876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3044.921833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3053.691330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3065.489066] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3086.002610] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 3097.065337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3097.078192] Lustre: lustre-MDT0000: reset Object Index mappings [ 3097.571912] LustreError: 114641:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3097.689841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3097.695086] Lustre: Skipped 1 previous similar message [ 3102.057906] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3110.991161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3111.021218] Lustre: lustre-MDT0001: reset Object Index mappings [ 3116.147405] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3119.238915] Lustre: 115791:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3127.620684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3135.413340] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3136.775286] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 3141.875352] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:381 to 0x280000401:418) [ 3143.848438] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3149.811209] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3149.828156] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3151.222956] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3160.830771] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3165.565747] Lustre: 117668:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3215.607613] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 18:54:31 (1789599271) [ 3241.952603] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3241.959878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3241.973533] Lustre: Skipped 5 previous similar messages [ 3241.977711] Lustre: Skipped 2 previous similar messages [ 3248.026123] Lustre: server umount lustre-MDT0000 complete [ 3251.564415] LustreError: 115391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789599307 with bad export cookie 12727250792792147405 [ 3251.583532] LustreError: 115391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3251.587197] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3252.193432] LustreError: 117688:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3252.222945] LustreError: 117688:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 3258.099893] Lustre: server umount lustre-MDT0001 complete [ 3268.709991] Lustre: server umount lustre-OST0000 complete [ 3279.322704] Lustre: server umount lustre-OST0001 complete [ 3287.042270] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 3297.276831] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 3333.356806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3333.593870] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3333.629963] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3333.731278] Lustre: lustre-MDT0000: new disk, initializing [ 3333.813797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3333.826832] Lustre: Skipped 3 previous similar messages [ 3333.841738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3338.491767] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3346.568041] Lustre: server umount lustre-MDT0000 complete [ 3357.870370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3358.172525] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3358.209349] Lustre: 121936:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3358.225836] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3364.797258] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3370.439703] Lustre: server umount lustre-MDT0000 complete [ 3380.677878] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 18:57:15 (1789599435) [ 3382.717768] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3384.748651] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 18:57:20 (1789599440) [ 3404.940106] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 3415.599187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3420.630318] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3429.121128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3429.233133] Lustre: 123986:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3429.288629] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3429.292808] Lustre: Skipped 1 previous similar message [ 3429.361423] Lustre: lustre-MDT0001: new disk, initializing [ 3429.495323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3429.509086] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3434.043784] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3439.259234] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3446.460896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3446.854364] Lustre: lustre-OST0000: new disk, initializing [ 3446.862378] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3446.881522] Lustre: 125567:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3448.716624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3448.732566] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3448.822388] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3453.044662] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3464.371325] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3464.621580] Lustre: lustre-OST0001: new disk, initializing [ 3464.625396] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3464.637829] Lustre: 126436:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3466.623188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3466.637582] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3466.798858] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3471.606993] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3481.042090] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3485.202718] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3884.509994] Lustre: 132872:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3884.519353] Lustre: 132872:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3884.523386] Lustre: 132872:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3884.528338] Lustre: 132872:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3884.533806] Lustre: 132872:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3884.541964] Lustre: 132872:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4168.433251] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 19:10:23 (1789600223) [ 4187.104568] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4187.113412] LustreError: Skipped 1 previous similar message [ 4187.120262] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4187.133615] Lustre: Skipped 2 previous similar messages [ 4187.141961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4187.149893] Lustre: Skipped 9 previous similar messages [ 4190.063689] Lustre: server umount lustre-MDT0000 complete [ 4193.732332] LustreError: 132884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789600249 with bad export cookie 12727250792792215578 [ 4193.735309] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4193.759448] LustreError: 132884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4193.764642] LustreError: 123995:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4193.768111] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4193.768119] Lustre: Skipped 3 previous similar messages [ 4193.820502] LustreError: 123995:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4198.880878] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4198.896729] Lustre: Skipped 7 previous similar messages [ 4200.459032] Lustre: server umount lustre-MDT0001 complete [ 4218.338608] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4218.474773] Lustre: server umount lustre-OST0000 complete [ 4236.256042] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4236.532831] Lustre: server umount lustre-OST0001 complete [ 4245.159351] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 4256.719835] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 4287.889732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4288.217521] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4288.260480] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4288.368564] Lustre: lustre-MDT0000: new disk, initializing [ 4288.444381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4288.448953] Lustre: Skipped 5 previous similar messages [ 4288.461761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4293.407427] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4303.283672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4303.348692] Lustre: 135927:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4303.394181] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4303.397742] Lustre: Skipped 1 previous similar message [ 4303.449463] Lustre: lustre-MDT0001: new disk, initializing [ 4303.559160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4303.576476] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4307.366325] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4316.202261] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4321.554876] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4329.223572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4329.618845] Lustre: lustre-OST0000: new disk, initializing [ 4329.623991] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4329.633718] Lustre: 137893:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4329.734264] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4329.740679] Lustre: Skipped 1 previous similar message [ 4331.092701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4331.102078] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4331.205896] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4335.723895] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4345.730672] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4356.470542] Lustre: lustre-MDD0000: changelog on [ 4360.618149] Lustre: lustre-MDD0001: changelog on [ 6271.473848] Lustre: lustre-MDD0001: changelog off [ 6273.768309] Lustre: 135934:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6274.615248] Lustre: lustre-MDD0000: changelog off [ 6282.226721] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6282.238978] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6282.240115] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6282.250613] Lustre: Skipped 2 previous similar messages [ 6286.317943] Lustre: server umount lustre-OST0000 complete [ 6290.406219] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6290.422292] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6290.451448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6290.460239] Lustre: Skipped 1 previous similar message [ 6292.450624] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6295.797608] Lustre: server umount lustre-MDT0000 complete [ 6297.569195] LustreError: 135939:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6297.582679] LustreError: 135939:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 6298.997653] LustreError: 135919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789602355 with bad export cookie 12727250792793040829 [ 6299.020404] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6299.380521] Lustre: server umount lustre-MDT0001 complete [ 6312.462098] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 6315.651822] Key type lgssc unregistered [ 6316.055137] LNet: 147031:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6316.061334] LNetError: 147031:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6317.101630] LNet: Removed LNI 192.168.206.155@tcp [ 6318.296505] Key type .llcrypt unregistered [ 6318.302135] Key type ._llcrypt unregistered [ 6338.755735] Key type ._llcrypt registered [ 6338.765729] Key type .llcrypt registered [ 6338.915393] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 6354.782803] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 6355.875781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6355.963831] alg: No test for adler32 (adler32-zlib) [ 6357.091295] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [ 6357.397773] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 6359.071403] Key type lgssc registered [ 6360.110669] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6396.215150] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6396.238441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6397.499125] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6397.522376] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6397.589598] Lustre: lustre-MDT0000: new disk, initializing [ 6397.668421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6397.689868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6402.307769] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6411.992238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6412.091571] Lustre: 150228:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6412.149596] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6412.152908] Lustre: Skipped 1 previous similar message [ 6412.208036] Lustre: lustre-MDT0001: new disk, initializing [ 6412.291290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6412.332577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6412.347414] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6416.342346] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6425.784846] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6432.266795] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6439.678323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6440.026529] Lustre: lustre-OST0000: new disk, initializing [ 6440.031227] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6440.042963] Lustre: 152198:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6440.185737] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6445.142124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6445.152058] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6445.301117] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6446.032542] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6456.998339] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6463.533681] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_module obdecho/obdecho [ 6465.323718] LustreError: 153204:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6465.329222] LustreError: 153204:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6467.194992] LustreError: 153300:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6478.817559] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6478.828202] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6478.855249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6481.376132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6481.377981] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6482.881286] Lustre: server umount lustre-MDT0000 complete [ 6485.952751] LustreError: 150221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789602542 with bad export cookie 3578009398577866072 [ 6485.953950] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6485.971263] LustreError: 150221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6486.258809] Lustre: server umount lustre-MDT0001 complete [ 6500.705089] Lustre: server umount lustre-OST0000 complete [ 6520.221775] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 6531.224972] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 6569.769850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6570.019825] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6570.049387] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6570.121910] Lustre: lustre-MDT0000: new disk, initializing [ 6570.241657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6570.252563] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6574.719815] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6584.952926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6585.108925] Lustre: 157018:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6585.204869] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6585.209179] Lustre: Skipped 1 previous similar message [ 6585.280943] Lustre: lustre-MDT0001: new disk, initializing [ 6585.382110] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6585.425340] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6585.436736] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6589.755866] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6598.319495] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6603.737017] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6610.458395] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6610.829309] Lustre: lustre-OST0000: new disk, initializing [ 6610.832996] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6610.845779] Lustre: 158986:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6610.974933] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6617.968867] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6618.701546] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6618.717443] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6618.839240] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6627.461546] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6631.577539] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6631.806911] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6636.621392] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6636.838885] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6639.585859] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6639.589147] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6639.593263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6639.593271] Lustre: Skipped 2 previous similar messages [ 6639.611243] Lustre: Skipped 2 previous similar messages [ 6639.644987] LustreError: Skipped 1 previous similar message [ 6644.707911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6644.716847] Lustre: Skipped 2 previous similar messages [ 6645.151891] Lustre: server umount lustre-OST0000 complete [ 6649.825518] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6649.849989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6649.854723] Lustre: Skipped 1 previous similar message [ 6660.066907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6660.076142] Lustre: Skipped 3 previous similar messages [ 6663.135435] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6663.478936] Lustre: server umount lustre-MDT0000 complete [ 6667.007082] LustreError: 157007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789602723 with bad export cookie 3578009398577867031 [ 6667.028191] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6667.342979] Lustre: server umount lustre-MDT0001 complete [ 6670.785486] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 19:52:06 (1789602726) [ 6672.223151] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6673.748919] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 19:52:09 (1789602729) [ 6680.510613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6681.074544] LustreError: 161401:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6681.099679] LustreError: 161401:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 6681.150841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6685.306453] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6686.179351] LustreError: 161402:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6691.299672] LustreError: 161401:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6693.283342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6698.382656] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6705.474850] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6710.732732] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6716.159774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6716.395927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6716.403527] Lustre: Skipped 1 previous similar message [ 6721.411036] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6728.464101] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6734.426214] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.55@tcp (stopping) [ 6734.816352] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6734.832314] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6734.853227] Lustre: Skipped 1 previous similar message [ 6735.911191] Lustre: server umount lustre-MDT0000 complete [ 6736.884842] LustreError: 163446:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6736.908290] LustreError: 163446:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6739.736798] LustreError: 161382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789602795 with bad export cookie 3578009398577867906 [ 6739.737667] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6739.749734] LustreError: 161382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6741.985538] LustreError: 162416:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6742.019692] LustreError: 162416:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6750.118615] LustreError: 162416:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6750.132110] LustreError: 162416:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 6754.271152] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6754.572910] Lustre: server umount lustre-MDT0001 complete [ 6760.204399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6771.789559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6772.085294] LustreError: 165242:0:(llog.c:1655:llog_backup()) MGC192.168.206.155@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6772.090236] Lustre: 165242:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.155@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6775.376202] LustreError: 165247:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6810.594330] LustreError: 165249:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6810.613284] LustreError: 165249:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 26 previous similar messages [ 6832.607349] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6832.611686] Lustre: 165242:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.155@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6873.606714] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6876.759474] LustreError: 165248:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6876.783581] LustreError: 165248:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 44 previous similar messages [ 6878.550686] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6878.699532] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6886.661776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6901.229934] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 6911.466567] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x31a7a497453e45e2 [ 6911.502350] Lustre: MGC192.168.206.155@tcp: Connection restored to 0@lo (at 0@lo) [ 6916.374967] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6917.102607] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6923.195517] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6924.448373] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6930.055793] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6931.738904] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6952.930967] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6952.933922] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6952.938124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6952.938132] Lustre: Skipped 6 previous similar messages [ 6952.969780] Lustre: Skipped 5 previous similar messages [ 6954.175452] Lustre: server umount lustre-MDT0000 complete [ 6957.379254] LustreError: 161383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789603013 with bad export cookie 3578009398577874402 [ 6957.391400] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6957.745922] Lustre: server umount lustre-MDT0001 complete [ 6971.537581] Lustre: server umount lustre-OST0000 complete [ 6993.371848] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 7004.001643] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 7039.614683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7039.778311] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7039.799722] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7039.872453] Lustre: lustre-MDT0000: new disk, initializing [ 7039.997384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7040.010263] Lustre: Skipped 1 previous similar message [ 7040.032751] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7044.239138] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7054.458652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7054.590498] Lustre: 170350:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7054.865177] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7059.818949] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7069.418229] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7074.648680] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7080.737592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7081.063083] Lustre: lustre-OST0000: new disk, initializing [ 7081.066373] Lustre: Skipped 1 previous similar message [ 7081.069587] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7081.074571] Lustre: Skipped 2 previous similar messages [ 7081.079368] Lustre: 172319:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7086.676109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7086.685687] Lustre: Skipped 1 previous similar message [ 7086.693141] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7086.780705] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7087.113614] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7095.654640] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7100.746758] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7100.922546] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7105.072227] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7105.253712] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7107.560379] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7107.562824] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7107.568734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7107.568742] Lustre: Skipped 2 previous similar messages [ 7107.572651] LustreError: Skipped 1 previous similar message [ 7107.600079] Lustre: Skipped 1 previous similar message [ 7121.375287] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7121.587105] Lustre: server umount lustre-OST0000 complete [ 7126.496130] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7139.295422] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7139.560948] Lustre: server umount lustre-MDT0000 complete [ 7143.093881] LustreError: 170342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789603199 with bad export cookie 3578009398577891195 [ 7143.096866] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7143.104865] LustreError: 170342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 7143.463619] Lustre: server umount lustre-MDT0001 complete [ 7151.609948] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 20:00:07 (1789603207) [ 7157.842289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7158.319115] LustreError: 174730:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7158.360769] LustreError: 174730:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 31 previous similar messages [ 7162.791772] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7170.835613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7175.540883] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7182.725121] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7187.868451] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7193.013542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7198.736847] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7205.753794] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7215.238462] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 20:01:10 (1789603270) [ 7219.170638] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7219.189300] Lustre: Skipped 3 previous similar messages [ 7219.200675] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7224.792805] Lustre: server umount lustre-OST0000 complete [ 7235.072825] Lustre: server umount lustre-MDT0000 complete [ 7238.480700] LustreError: 174712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789603294 with bad export cookie 3578009398577892084 [ 7238.498795] LustreError: 174712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7238.501577] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7238.880110] Lustre: server umount lustre-MDT0001 complete [ 7250.779934] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 7254.009381] Key type lgssc unregistered [ 7254.360379] LNet: 178467:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7254.391455] LNetError: 178467:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7254.417291] LNet: Removed LNI 192.168.206.155@tcp [ 7255.467645] Key type .llcrypt unregistered [ 7255.471056] Key type ._llcrypt unregistered [ 7274.538360] Key type ._llcrypt registered [ 7274.539841] Key type .llcrypt registered [ 7274.703087] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 7275.893255] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7275.940459] alg: No test for adler32 (adler32-zlib) [ 7276.978680] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [ 7277.225105] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 7278.927189] Key type lgssc registered [ 7280.142776] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7288.042081] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7288.079352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7303.711714] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7308.831675] LustreError: 179709:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.155@tcp: failed processing log, type 4: rc = -110 [ 7313.953144] LustreError: 179709:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7313.967139] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7313.981260] LustreError: MGC192.168.206.155@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7313.995947] LustreError: 179709:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7314.007474] LustreError: 179709:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -22 [ 7314.009973] LustreError: 179709:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 7314.055989] Lustre: server umount lustre-OST0000 complete [ 7314.060923] LustreError: 179709:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7321.263549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7322.698329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7327.513581] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7331.811726] LustreError: 180209:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7331.828886] LustreError: 180209:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7336.045308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7336.480727] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7340.462633] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7347.299919] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7351.978644] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7357.514554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7357.717515] LustreError: 182024:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7357.735121] Lustre: 182024:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.206.155@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7357.945162] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7363.707904] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7370.900398] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7377.376148] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7377.383132] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7377.385623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7377.404842] Lustre: Skipped 1 previous similar message [ 7377.411123] Lustre: Skipped 2 previous similar messages [ 7379.536708] Lustre: server umount lustre-MDT0000 complete [ 7382.505553] LustreError: 180208:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7382.524133] LustreError: 180208:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7383.042511] LustreError: 180944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789603439 with bad export cookie 13664978479719826515 [ 7383.045075] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7383.051849] LustreError: 180944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7383.332200] Lustre: server umount lustre-MDT0001 complete [ 7396.075782] Lustre: server umount lustre-OST0000 complete [ 7405.294467] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 7414.753538] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 7454.337049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7454.635119] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7454.663816] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7454.801345] Lustre: lustre-MDT0000: new disk, initializing [ 7454.944255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7454.977943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7459.360796] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7470.092090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7470.267817] Lustre: 185551:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7470.322432] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7470.325565] Lustre: Skipped 1 previous similar message [ 7470.398144] Lustre: lustre-MDT0001: new disk, initializing [ 7470.537489] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7470.597523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7470.611401] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7475.285281] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7485.347373] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7492.204639] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7499.417350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7499.800881] Lustre: lustre-OST0000: new disk, initializing [ 7499.804238] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7499.810838] Lustre: 187519:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7499.873437] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7506.567751] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7506.992941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7507.009816] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7507.135629] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7516.643098] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7521.616382] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7521.922561] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7526.881108] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7527.157322] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7533.025196] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7533.046684] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7533.047612] LustreError: Skipped 1 previous similar message [ 7533.048369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7533.071656] Lustre: Skipped 2 previous similar messages [ 7533.115678] Lustre: Skipped 1 previous similar message [ 7535.551498] Lustre: server umount lustre-OST0000 complete [ 7542.242878] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7542.252056] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7542.266331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7546.248788] Lustre: server umount lustre-MDT0000 complete [ 7548.384771] LustreError: 185557:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7548.402807] LustreError: 185557:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7549.540867] LustreError: 185543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789603605 with bad export cookie 13664978479719827236 [ 7549.550179] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7549.553342] LustreError: 185543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7549.895685] Lustre: server umount lustre-MDT0001 complete [ 7558.231938] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 20:06:53 (1789603613) [ 7576.079630] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 7578.878479] Key type lgssc unregistered [ 7579.258877] LNet: 190463:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7579.266179] LNetError: 190463:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7579.291836] LNet: Removed LNI 192.168.206.155@tcp [ 7580.244229] Key type .llcrypt unregistered [ 7580.246042] Key type ._llcrypt unregistered [ 7598.180300] Key type ._llcrypt registered [ 7598.182959] Key type .llcrypt registered [ 7598.297629] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 7599.148951] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7599.333239] alg: No test for adler32 (adler32-zlib) [ 7600.505228] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [ 7600.846435] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 7602.599211] Key type lgssc registered [ 7603.916738] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7613.719289] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7613.752220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7615.441055] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7619.697331] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7625.709615] LustreError: 191679:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7625.724770] LustreError: 191679:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7628.428486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7628.931437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7633.681920] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7640.719953] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7646.911614] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7654.084051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7654.354633] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7654.363444] LustreError: 193546:0:(llog.c:1696:llog_backup()) MGC192.168.206.155@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7654.382371] Lustre: 193546:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.206.155@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7654.525109] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7660.467595] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7667.482144] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7669.729529] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7669.747156] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7669.777447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7671.269814] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7671.286538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7671.313863] Lustre: Skipped 1 previous similar message [ 7671.322215] Lustre: Skipped 2 previous similar messages [ 7674.404396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7674.416416] Lustre: Skipped 2 previous similar messages [ 7675.929275] Lustre: server umount lustre-MDT0000 complete [ 7679.457116] LustreError: 191680:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7679.473161] LustreError: 191680:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7679.486763] LustreError: 191659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789603735 with bad export cookie 10129027923987087037 [ 7679.490192] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7679.497022] LustreError: 191659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7680.607325] LustreError: 192411:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000006a1867eb x1876535444270592/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7680.746182] Lustre: server umount lustre-MDT0001 complete [ 7694.948587] Lustre: server umount lustre-OST0000 complete [ 7704.377143] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 7714.484361] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 7749.010767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7749.289174] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7749.332359] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7749.411860] Lustre: lustre-MDT0000: new disk, initializing [ 7749.518833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7749.535423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7754.540543] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7764.817715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7764.945363] Lustre: 197069:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7765.006094] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7765.008940] Lustre: Skipped 1 previous similar message [ 7765.012414] LustreError: 197851:0:(llog.c:1696:llog_backup()) MGC192.168.206.155@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7765.021184] LustreError: 197851:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7765.025857] Lustre: 197851:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.155@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7765.045135] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7765.053273] Lustre: Skipped 1 previous similar message [ 7765.140429] Lustre: lustre-MDT0001: new disk, initializing [ 7765.244911] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7765.299889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7765.316981] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7769.518472] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7778.506668] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7784.522299] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7791.647287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7791.992917] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7791.997906] Lustre: Skipped 1 previous similar message [ 7792.004182] LustreError: 199014:0:(llog.c:1696:llog_backup()) MGC192.168.206.155@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7792.013498] LustreError: 199014:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7792.022678] Lustre: 199014:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.155@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7792.035964] Lustre: 199014:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7792.058603] Lustre: lustre-OST0000: new disk, initializing [ 7792.065575] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7792.074435] Lustre: 199037:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7792.182792] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7799.109236] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7801.912407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7801.919905] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7802.052584] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7809.210750] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7814.492291] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7814.764194] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7820.086169] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7820.343617] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7822.816909] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7822.819493] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7822.823051] LustreError: Skipped 1 previous similar message [ 7822.835451] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7822.835966] Lustre: Skipped 1 previous similar message [ 7822.859422] Lustre: Skipped 1 previous similar message [ 7827.944589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7827.956479] Lustre: Skipped 1 previous similar message [ 7828.552101] Lustre: server umount lustre-OST0000 complete [ 7833.058672] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7833.071235] Lustre: Skipped 1 previous similar message [ 7838.190136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7838.198224] Lustre: Skipped 3 previous similar messages [ 7839.000130] Lustre: server umount lustre-MDT0000 complete [ 7843.296511] LustreError: 197076:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7843.337715] LustreError: 197076:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7843.445096] LustreError: 197863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789603899 with bad export cookie 10129027923987087765 [ 7843.461657] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7843.469026] LustreError: 197863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7844.083740] Lustre: server umount lustre-MDT0001 complete [ 7853.995956] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 20:11:49 (1789603909) [ 7871.238766] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 7881.552059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7881.895489] LustreError: 202164:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7881.941568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7886.989546] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7887.331719] LustreError: 202165:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7896.744135] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7901.545855] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7904.423235] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7910.540270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7918.452848] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7928.343191] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7928.600128] Lustre: lustre-OST0001: new disk, initializing [ 7928.610877] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7928.617358] Lustre: 204471:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7928.705278] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7928.714267] Lustre: Skipped 2 previous similar messages [ 7934.813634] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7935.650106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7935.691256] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7935.867900] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7946.082225] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7957.979467] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7960.259726] Lustre: DEBUG MARKER: ADD OST3 [ 7965.835985] loop: module loaded [ 7965.942830] loop0: detected capacity change from 0 to 204800000 [ 7966.134946] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7966.154558] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7966.180628] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7972.823778] loop0: detected capacity change from 0 to 204800000 [ 7973.412059] Lustre: DEBUG MARKER: STOP OST3 [ 7979.310832] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7979.568917] Lustre: lustre-OST0003: new disk, initializing [ 7979.573884] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7979.585565] Lustre: 206500:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7981.455425] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7981.660872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7981.670550] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7981.712551] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7982.908218] Lustre: Failing over lustre-OST0003 [ 7983.045389] Lustre: server umount lustre-OST0003 complete [ 7983.524209] LustreError: 206701:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7985.333648] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7986.309787] LustreError: 203652:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.206.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7986.339040] LustreError: 203652:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7986.663597] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7986.670832] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7994.283560] Lustre: DEBUG MARKER: START OST3 again [ 8002.070204] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8002.875624] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8002.894074] Lustre: Skipped 1 previous similar message [ 8002.927607] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8004.022060] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8004.505758] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8004.541474] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8009.591617] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8031.006845] Lustre: Failing over lustre-OST0003 [ 8031.244749] Lustre: server umount lustre-OST0003 complete [ 8033.766867] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8033.785548] LustreError: 203650:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8033.794608] Lustre: Skipped 2 previous similar messages [ 8033.813955] LustreError: 203650:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 12 previous similar messages [ 8049.132922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8049.140387] Lustre: Skipped 2 previous similar messages [ 8054.825312] Lustre: server umount lustre-MDT0000 complete [ 8058.490414] LustreError: 205242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789604114 with bad export cookie 10129027923987088647 [ 8058.496915] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8058.507501] LustreError: 205242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8059.361752] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8059.380299] Lustre: Skipped 5 previous similar messages [ 8064.956576] Lustre: server umount lustre-MDT0001 complete [ 8068.638693] Lustre: server umount lustre-OST0000 complete [ 8072.475672] Lustre: server umount lustre-OST0001 complete [ 8079.466197] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 8088.053867] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 8125.370373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8125.682915] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8125.706257] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8125.763254] Lustre: lustre-MDT0000: new disk, initializing [ 8126.009613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8131.590536] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8142.718292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8142.795366] Lustre: 211936:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8142.905237] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8142.910245] Lustre: Skipped 1 previous similar message [ 8143.031474] Lustre: lustre-MDT0001: new disk, initializing [ 8143.207912] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8143.218141] Lustre: Skipped 1 previous similar message [ 8143.381247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8143.406565] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8148.988103] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8158.823169] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8164.604342] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8171.787588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8172.066567] Lustre: 213907:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8173.373723] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8173.480260] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8179.219084] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8188.237678] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8193.847574] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8194.113752] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8200.592422] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8200.722537] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8204.256264] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8204.268510] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8204.276355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8204.279860] Lustre: Skipped 1 previous similar message [ 8204.293984] Lustre: Skipped 9 previous similar messages [ 8208.827516] Lustre: server umount lustre-OST0000 complete [ 8218.941159] Lustre: server umount lustre-MDT0000 complete [ 8219.622107] LustreError: 211948:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8219.635415] LustreError: 211948:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 8222.443897] LustreError: 211928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789604278 with bad export cookie 10129027923987090943 [ 8222.447826] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8222.454704] LustreError: 211928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8222.800690] Lustre: server umount lustre-MDT0001 complete [ 8226.082719] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 20:18:01 (1789604281) [ 8242.172202] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 8253.386103] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 8289.263411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8289.518199] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8289.553099] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8289.561088] Lustre: Skipped 1 previous similar message [ 8289.666900] Lustre: lustre-MDT0000: new disk, initializing [ 8289.671102] Lustre: Skipped 1 previous similar message [ 8289.753420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8289.758748] Lustre: Skipped 1 previous similar message [ 8294.621194] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8303.793119] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8303.927751] Lustre: 218517:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8304.245127] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8308.681368] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8318.664060] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8324.398111] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8332.245108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8332.594374] Lustre: 220486:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8333.861824] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8339.418368] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8350.665816] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8355.178640] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8355.426408] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8360.926278] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8361.228590] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8364.512505] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8364.521397] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8364.530758] LustreError: Skipped 2 previous similar messages [ 8364.535491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8364.555090] Lustre: Skipped 3 previous similar messages [ 8364.580971] Lustre: Skipped 3 previous similar messages [ 8369.640591] LustreError: 220471:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8369.660035] LustreError: 220471:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8369.737232] Lustre: server umount lustre-OST0000 complete [ 8379.824059] Lustre: server umount lustre-MDT0000 complete [ 8383.218975] LustreError: 219321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789604439 with bad export cookie 10129027923987091825 [ 8383.224573] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8383.236785] LustreError: 219321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8383.652876] Lustre: server umount lustre-MDT0001 complete [ 8391.522555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8395.635698] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8402.524995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8403.022016] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8403.028942] Lustre: Skipped 5 previous similar messages [ 8408.912875] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8417.288603] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8423.887875] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8430.506270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8436.280256] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8443.311928] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8455.864350] Lustre: server umount lustre-OST0000 complete [ 8466.496790] Lustre: server umount lustre-MDT0000 complete [ 8470.330874] LustreError: 222685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789604526 with bad export cookie 10129027923987092707 [ 8470.346298] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8470.351368] LustreError: 222685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8470.767124] Lustre: server umount lustre-MDT0001 complete [ 8582.495773] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 8585.564496] Key type lgssc unregistered [ 8585.887065] LNet: 226368:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8585.897639] LNetError: 226368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8585.936169] LNet: Removed LNI 192.168.206.155@tcp [ 8587.053147] Key type .llcrypt unregistered [ 8587.057743] Key type ._llcrypt unregistered [ 8611.485553] Key type ._llcrypt registered [ 8611.487646] Key type .llcrypt registered [ 8611.576189] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 8630.363405] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 8631.920079] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8632.148996] alg: No test for adler32 (adler32-zlib) [ 8633.468794] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [ 8633.805459] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 8635.487214] Key type lgssc registered [ 8637.035866] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8674.453402] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8674.480774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8675.679364] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8675.697806] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8675.746546] Lustre: lustre-MDT0000: new disk, initializing [ 8675.842739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8675.863925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8681.313592] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8693.220681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8693.388470] Lustre: 229719:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8693.465918] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8693.483506] Lustre: Skipped 1 previous similar message [ 8693.610408] Lustre: lustre-MDT0001: new disk, initializing [ 8693.764688] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8693.812200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8693.832268] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8699.064695] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8708.396293] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8714.566459] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8721.749759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8721.994732] Lustre: lustre-OST0000: new disk, initializing [ 8722.003496] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8722.010321] Lustre: 231689:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8722.106700] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8727.239814] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8729.706351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8729.713428] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8729.878290] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8736.150809] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8741.269979] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8741.571273] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8746.848695] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8747.070981] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8750.563732] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8750.570078] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8750.575577] LustreError: Skipped 1 previous similar message [ 8750.576829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8750.590565] Lustre: Skipped 1 previous similar message [ 8750.606201] Lustre: Skipped 1 previous similar message [ 8755.689692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8760.801325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8760.817334] Lustre: Skipped 2 previous similar messages [ 8763.360065] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8763.661334] Lustre: server umount lustre-OST0000 complete [ 8770.529275] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8770.538079] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8770.556814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8774.050691] Lustre: server umount lustre-MDT0000 complete [ 8778.115585] LustreError: 229709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789604834 with bad export cookie 9648997748085282517 [ 8778.117285] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8778.141267] LustreError: 229709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8778.733512] Lustre: server umount lustre-MDT0001 complete [ 8800.033515] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 8809.319744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8809.701096] LustreError: 234624:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8809.740019] LustreError: 234624:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8809.762135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8813.354751] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8815.082511] LustreError: 234625:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8819.175250] LustreError: 234624:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8821.911872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8822.389891] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8827.228906] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8830.066757] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8836.654895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8837.158340] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8843.870923] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8852.825214] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8853.114094] Lustre: lustre-OST0001: new disk, initializing [ 8853.121763] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8853.129394] Lustre: 236930:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8853.242765] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8858.925323] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8859.711263] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8859.724265] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8859.817175] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8870.398914] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8877.133905] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8890.479808] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 20:29:06 (1789604946) [ 8893.921625] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8893.932296] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8893.941347] Lustre: Skipped 1 previous similar message [ 8893.947040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8893.959247] Lustre: Skipped 2 previous similar messages [ 8895.973361] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8898.702779] Lustre: server umount lustre-MDT0000 complete [ 8901.089442] LustreError: 234624:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8901.124453] LustreError: 234624:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8902.132686] LustreError: 236929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789604958 with bad export cookie 9648997748085283399 [ 8902.138074] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8902.141932] LustreError: 236929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8902.487657] Lustre: server umount lustre-MDT0001 complete [ 8916.500591] Lustre: server umount lustre-OST0000 complete [ 8930.353921] Lustre: server umount lustre-OST0001 complete [ 8936.897593] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 8947.326768] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 8981.828939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8982.090724] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8982.121105] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8982.211192] Lustre: lustre-MDT0000: new disk, initializing [ 8982.315165] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8982.371937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8986.814713] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8997.006704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8997.110223] Lustre: 240879:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8997.160061] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8997.163929] Lustre: Skipped 1 previous similar message [ 8997.230745] Lustre: lustre-MDT0001: new disk, initializing [ 8997.378835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8997.386633] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9002.029957] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9011.264405] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9017.203600] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9023.628244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9023.861410] Lustre: lustre-OST0000: new disk, initializing [ 9023.865241] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9023.870415] Lustre: 242845:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9025.920735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9025.942953] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9026.102518] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9030.778781] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9040.317946] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9044.684814] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9044.901625] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9049.112292] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9049.338073] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9051.617202] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9051.624941] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9051.637447] Lustre: Skipped 2 previous similar messages [ 9051.647383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9051.661441] Lustre: Skipped 5 previous similar messages [ 9057.435875] Lustre: server umount lustre-OST0000 complete [ 9061.857795] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9061.873451] Lustre: Skipped 2 previous similar messages [ 9067.701869] Lustre: server umount lustre-MDT0000 complete [ 9071.078697] LustreError: 240871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789605127 with bad export cookie 9648997748085285534 [ 9071.081658] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9071.089238] LustreError: 240871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9071.387613] Lustre: server umount lustre-MDT0001 complete [ 9079.389051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9079.852602] LustreError: 245062:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9079.934502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9079.942202] Lustre: Skipped 2 previous similar messages [ 9083.918405] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9090.532744] LustreError: 245062:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9090.559035] LustreError: 245062:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9092.057392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9096.389682] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9102.632800] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9107.351300] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9114.821633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9121.917549] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9129.064754] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9276.388281] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9276.397295] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9276.414123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9276.423567] Lustre: Skipped 6 previous similar messages [ 9290.207118] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9290.368695] Lustre: server umount lustre-OST0000 complete [ 9297.380348] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9300.008113] Lustre: server umount lustre-MDT0000 complete [ 9303.008944] LustreError: 245059:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9303.023344] LustreError: 245059:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9303.480412] LustreError: 245043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789605359 with bad export cookie 9648997748085286416 [ 9303.490545] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9303.496050] LustreError: 245043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9303.933914] Lustre: server umount lustre-MDT0001 complete [ 9395.165184] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 9397.409039] Key type lgssc unregistered [ 9397.679370] LNet: 248670:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9397.684262] LNetError: 248670:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9397.704067] LNet: Removed LNI 192.168.206.155@tcp [ 9398.377909] Key type .llcrypt unregistered [ 9398.382583] Key type ._llcrypt unregistered [ 9420.553874] Key type ._llcrypt registered [ 9420.557404] Key type .llcrypt registered [ 9420.688119] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 9434.675545] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 9435.772882] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9435.812101] alg: No test for adler32 (adler32-zlib) [ 9436.941648] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [ 9437.203545] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 9438.839208] Key type lgssc registered [ 9439.785906] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9478.338916] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 9486.844370] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9486.864965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9488.193968] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9488.227340] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9488.322995] Lustre: lustre-MDT0000: new disk, initializing [ 9488.405381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9488.424759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9492.724307] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9502.810130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9502.978269] Lustre: 252735:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9503.057130] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9503.063956] Lustre: Skipped 1 previous similar message [ 9503.151123] Lustre: lustre-MDT0001: new disk, initializing [ 9503.260912] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9503.290818] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9503.311290] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9507.908690] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9512.817329] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9519.976923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9520.376332] Lustre: lustre-OST0000: new disk, initializing [ 9520.384335] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9520.394652] Lustre: 254367:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9520.553591] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9524.442126] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9524.455057] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9524.601435] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9527.388384] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9537.872333] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9538.094655] Lustre: lustre-OST0001: new disk, initializing [ 9538.098716] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9538.107379] Lustre: 255236:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9538.188567] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9544.405475] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9546.358759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9546.376776] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9546.488776] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9555.345913] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9562.227612] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9570.991749] Lustre: server umount lustre-MDT0000 complete [ 9572.322870] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9572.349895] Lustre: Skipped 2 previous similar messages [ 9574.304630] LustreError: 256005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789605630 with bad export cookie 6571990018590148125 [ 9574.306739] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9574.311993] LustreError: 256005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9576.637693] Lustre: server umount lustre-MDT0001 complete [ 9590.243790] Lustre: server umount lustre-OST0000 complete [ 9592.800120] Lustre: 250242:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789605633/real 1789605633] req@ffff9210c78a5180 x1876537369502080/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789605649 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9592.833624] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9592.856686] Lustre: Skipped 1 previous similar message [ 9593.817418] Lustre: server umount lustre-OST0001 complete [ 9606.925520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9615.970183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9623.976259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9633.072081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9648.784629] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 9658.511835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9658.542746] Lustre: lustre-MDT0000: reset Object Index mappings [ 9658.967798] LustreError: 260244:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9658.984294] LustreError: 260244:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 9659.034335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9663.334731] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9664.483597] LustreError: 260245:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9664.507980] LustreError: 260245:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9669.600827] LustreError: 260244:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9671.497244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9671.524863] Lustre: lustre-MDT0001: reset Object Index mappings [ 9672.047339] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9676.545874] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9679.541829] Lustre: 261385:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9686.142974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9692.363582] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9697.787511] LustreError: 261739:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9700.884752] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9701.104488] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9701.110706] Lustre: Skipped 1 previous similar message [ 9704.184320] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9704.200071] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9706.898974] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9713.466593] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9716.303421] Lustre: 263257:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9719.779752] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9719.781973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9719.792330] Lustre: Skipped 3 previous similar messages [ 9719.799580] Lustre: Skipped 3 previous similar messages [ 9724.903453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9726.003582] Lustre: server umount lustre-MDT0000 complete [ 9729.285731] LustreError: 260225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789605785 with bad export cookie 6571990018590154915 [ 9729.293535] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9729.296193] LustreError: 260225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9729.572905] Lustre: server umount lustre-MDT0001 complete [ 9741.884342] Lustre: server umount lustre-OST0000 complete [ 9755.343443] Lustre: server umount lustre-OST0001 complete [ 9772.270700] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 20:43:47 (1789605827) [ 9787.313066] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 9796.922761] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 9830.113633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9830.333875] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9830.353231] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9830.419328] Lustre: lustre-MDT0000: new disk, initializing [ 9830.499067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9830.516489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9833.836106] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9843.127596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9843.237685] Lustre: 267513:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9843.281987] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9843.285051] Lustre: Skipped 1 previous similar message [ 9843.332574] Lustre: lustre-MDT0001: new disk, initializing [ 9843.422848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9843.432876] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9847.189128] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9855.356561] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9860.350739] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9866.451864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9866.790770] Lustre: lustre-OST0000: new disk, initializing [ 9866.798921] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9866.809615] Lustre: 269481:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9868.743692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9868.768379] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9868.935527] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9873.512572] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9882.064522] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9886.209248] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9886.403399] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9890.024862] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9890.219451] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9894.378703] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9894.380043] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9894.397481] LustreError: Skipped 1 previous similar message [ 9894.398469] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9894.398474] Lustre: Skipped 3 previous similar messages [ 9894.434902] Lustre: Skipped 1 previous similar message [ 9899.496457] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9899.504454] Lustre: Skipped 2 previous similar messages [ 9904.612525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9906.143145] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9906.266455] Lustre: server umount lustre-OST0000 complete [ 9909.738428] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9909.765579] Lustre: Skipped 1 previous similar message [ 9914.850597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9914.860796] Lustre: Skipped 4 previous similar messages [ 9915.550380] Lustre: server umount lustre-MDT0000 complete [ 9919.687590] LustreError: 269480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789605975 with bad export cookie 6571990018590157008 [ 9919.691512] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9919.695867] LustreError: 269480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9920.104881] Lustre: server umount lustre-MDT0001 complete [ 9936.087529] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 9943.681598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9943.936983] LustreError: 272416:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9943.967471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9943.973941] Lustre: Skipped 2 previous similar messages [ 9947.414924] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9954.274983] LustreError: 272417:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9954.298121] LustreError: 272417:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9955.028153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9959.752550] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9962.008277] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9966.473180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9971.338399] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9979.785191] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9980.003414] Lustre: lustre-OST0001: new disk, initializing [ 9980.012518] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9980.042977] Lustre: 274722:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9981.804780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9981.826486] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9981.978676] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9986.352528] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9994.728361] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9997.168167] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10001.335943] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10001.374148] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10011.617555] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10011.624212] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10011.632554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10013.771821] Lustre: server umount lustre-MDT0000 complete [10017.275966] LustreError: 272398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789606073 with bad export cookie 6571990018590157890 [10017.278457] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10017.287025] LustreError: 272398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10017.639971] Lustre: server umount lustre-MDT0001 complete [10031.805226] Lustre: server umount lustre-OST0000 complete [10034.111887] Lustre: 250244:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789606074/real 1789606074] req@ffff9210c8a8e680 x1876537369647616/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789606090 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10034.163485] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10034.179794] Lustre: Skipped 3 previous similar messages [10035.254651] Lustre: server umount lustre-OST0001 complete [10051.461087] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [10060.634828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10061.063194] LustreError: 278061:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10065.633920] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10073.707954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10074.262435] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10074.270130] Lustre: Skipped 4 previous similar messages [10078.870417] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10081.748480] Lustre: 279202:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10088.201547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10095.088567] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10099.754440] LustreError: 279559:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10099.784021] LustreError: 279559:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10103.794637] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10106.099737] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10106.114895] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10109.535980] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10116.967824] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10120.546843] Lustre: 281084:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10125.668557] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10125.909187] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10136.617454] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 20:49:52 (1789606192) [10150.881182] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10150.887527] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10150.899608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10150.904084] Lustre: Skipped 4 previous similar messages [10156.717120] Lustre: server umount lustre-MDT0000 complete [10159.383589] LustreError: 278042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789606215 with bad export cookie 6571990018590164295 [10159.392711] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10159.398109] LustreError: 278042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10164.705080] LustreError: 278058:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10164.731374] LustreError: 278058:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [10165.720334] Lustre: server umount lustre-MDT0001 complete [10175.096680] Lustre: server umount lustre-OST0000 complete [10178.375890] Lustre: server umount lustre-OST0001 complete [10184.523218] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [10193.204295] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [10236.208110] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [10246.459937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10246.680781] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10246.706849] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10246.781675] Lustre: lustre-MDT0000: new disk, initializing [10246.855499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10251.373275] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10261.199957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10261.374251] Lustre: 285608:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10261.662450] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10265.876056] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10276.511435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10276.666314] Lustre: 287241:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10278.098658] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10282.489875] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10295.261438] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10295.521693] Lustre: 288112:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10297.470583] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10297.477756] Lustre: Skipped 1 previous similar message [10297.600654] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10302.896569] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10313.020871] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10316.217538] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10316.221318] Lustre: Skipped 1 previous similar message [10330.341051] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 20:53:05 (1789606385) [10333.157763] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10333.166242] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10333.178115] Lustre: Skipped 5 previous similar messages [10333.188586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10333.192226] Lustre: Skipped 8 previous similar messages [10334.943622] Lustre: server umount lustre-MDT0000 complete [10338.372972] LustreError: 285600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789606394 with bad export cookie 6571990018590172261 [10338.379241] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10338.381360] LustreError: 285600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10338.738508] Lustre: server umount lustre-MDT0001 complete [10352.602979] Lustre: server umount lustre-OST0000 complete [10355.623070] Lustre: 250244:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789606395/real 1789606395] req@ffff9210f8a52680 x1876537369794304/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789606411 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10357.120321] Lustre: server umount lustre-OST0001 complete [10367.659742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10368.145342] LustreError: 290358:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10368.267759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10368.272928] Lustre: Skipped 6 previous similar messages [10373.854401] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10381.634195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10386.505860] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10393.793888] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10399.803761] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10407.153409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10414.167403] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10422.002762] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10441.557625] loop1: detected capacity change from 0 to 204800000 [10441.781731] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10441.796354] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10441.808506] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10447.227919] loop1: detected capacity change from 0 to 204800000 [10453.081345] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10453.156109] LustreError: 290346:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs12865: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10453.172697] LustreError: 290346:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs12865 [10453.179354] LustreError: 290346:0:(mgs_handler.c:517:mgs_target_reg()) fs12865-MDT0001: Error checking target index: rc = -13 [10453.185981] LustreError: fs12865-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10453.195892] LustreError: 294080:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [10453.208610] LustreError: 294080:0:(tgt_mount.c:2128:server_put_super()) no obd fs12865-MDT0001 [10453.213372] LustreError: 294080:0:(tgt_mount.c:116:server_deregister_mount()) fs12865-MDT0001 not registered [10453.248961] Lustre: server umount fs12865-MDT0001 complete [10453.251669] LustreError: 294080:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10461.714367] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10461.879859] Lustre: Setting parameter fs12865-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs12865-MDT0001 [10461.975684] Lustre: srv-fs12865-MDT0001: No data found on store. Initialize space. [10461.982547] Lustre: Skipped 4 previous similar messages [10462.106513] Lustre: fs12865-MDT0001: new disk, initializing [10462.111429] Lustre: Skipped 3 previous similar messages [10462.224355] Lustre: srv-fs12865-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10462.232879] Lustre: 294546:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs12865-MDT0001-mdtlov: can't create new llog: rc = -115 [10462.250096] LustreError: 294546:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs12865-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10466.868509] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10473.834294] Lustre: server umount fs12865-MDT0001 complete [10486.304986] loop1: detected capacity change from 0 to 204800000 [10491.797184] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10492.375337] Lustre: 296114:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs12865-MDT0001-mdtlov: can't create new llog: rc = -115 [10492.387560] LustreError: 296114:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs12865-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10493.937575] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10497.384633] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10502.395840] Lustre: server umount fs12865-MDT0001 complete [10509.287826] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10514.410742] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10514.416715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10514.427036] Lustre: Skipped 5 previous similar messages [10514.457235] Lustre: Skipped 5 previous similar messages [10517.671868] Lustre: server umount lustre-MDT0000 complete [10526.512177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10526.672784] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10530.757400] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10532.352036] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10538.683638] loop1: detected capacity change from 0 to 204800000 [10544.775855] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10545.457530] Lustre: 298565:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs12865-MDT0001-mdtlov: can't create new llog: rc = -115 [10545.478183] LustreError: 298565:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs12865-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10550.322286] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10554.593516] Lustre: server umount fs12865-MDT0001 complete [10575.850961] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10580.787813] Lustre: server umount lustre-OST0000 complete [10599.393059] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10599.585880] Lustre: server umount lustre-MDT0000 complete [10603.179417] LustreError: 290340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789606659 with bad export cookie 6571990018590174942 [10603.200477] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10603.633648] Lustre: server umount lustre-MDT0001 complete [10614.952525] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [10618.081704] Key type lgssc unregistered [10618.429741] LNet: 300908:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10618.438655] LNetError: 300908:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10618.471538] LNet: Removed LNI 192.168.206.155@tcp [10619.469430] Key type .llcrypt unregistered [10619.474585] Key type ._llcrypt unregistered [10640.525188] Key type ._llcrypt registered [10640.526813] Key type .llcrypt registered [10640.718981] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [10641.868216] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10641.994718] alg: No test for adler32 (adler32-zlib) [10643.162419] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [10643.448974] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [10645.279194] Key type lgssc registered [10646.458834] Lustre: Echo OBD driver; http://www.lustre.org/ [10654.096421] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10654.120982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10655.682098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10661.763986] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10664.931953] LustreError: 302178:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10664.961899] LustreError: 302178:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10670.051066] LustreError: 302177:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10671.597679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10672.019617] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10677.268889] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10684.140331] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10689.640266] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10696.874156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10697.361316] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10698.422329] LustreError: 304009:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10700.472597] LustreError: 304008:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10700.508411] LustreError: 304008:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10703.875819] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10705.895044] LustreError: 304008:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10705.921645] LustreError: 304008:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10711.137393] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10716.132742] LustreError: 304009:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10716.147124] LustreError: 304009:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [10728.357252] loop1: detected capacity change from 0 to 204800000 [10728.633574] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10728.662316] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10728.685376] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10734.159114] LustreError: 304010:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10734.196622] LustreError: 304010:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [10734.617737] loop1: detected capacity change from 0 to 204800000 [10740.588502] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10740.754378] Lustre: fs1922-OST0000: new disk, initializing [10740.765462] Lustre: srv-fs1922-OST0000: No data found on store. Initialize space. [10740.775452] Lustre: 305912:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10740.890915] Lustre: fs1922-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10747.362643] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10756.109677] Lustre: server umount fs1922-OST0000 complete [10767.345714] LustreError: 304087:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10767.391316] LustreError: 304087:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 19 previous similar messages [10770.348177] loop1: detected capacity change from 0 to 204800000 [10776.657488] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10777.004186] Lustre: fs1922-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10782.696919] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10782.779769] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10790.697096] loop2: detected capacity change from 0 to 204800000 [10790.824391] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10790.833752] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10790.843928] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10795.983584] loop2: detected capacity change from 0 to 204800000 [10801.390367] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10801.443701] LustreError: fs1922-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10801.460829] LustreError: fs1922-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10801.469619] LustreError: 308955:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [10801.476697] LustreError: 308955:0:(tgt_mount.c:2128:server_put_super()) no obd fs1922-OST0001 [10801.481681] LustreError: 308955:0:(tgt_mount.c:116:server_deregister_mount()) fs1922-OST0001 not registered [10801.509367] Lustre: server umount fs1922-OST0001 complete [10801.512667] LustreError: 308955:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10812.493409] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10812.618542] Lustre: fs1922-OST0001: new disk, initializing [10812.627788] Lustre: srv-fs1922-OST0001: No data found on store. Initialize space. [10812.642064] Lustre: 309517:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10812.701110] Lustre: fs1922-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10818.252136] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10826.396466] Lustre: server umount fs1922-OST0000 complete [10833.892351] LustreError: 304008:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10833.911027] LustreError: 304008:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 38 previous similar messages [10834.714955] Lustre: server umount fs1922-OST0001 complete [10846.462708] loop1: detected capacity change from 0 to 204800000 [10851.106054] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10851.338604] Lustre: fs1922-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10856.055909] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10861.805719] loop2: detected capacity change from 0 to 204800000 [10866.362912] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10872.347743] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10881.317740] Lustre: server umount fs1922-OST0000 complete [10890.512252] Lustre: server umount fs1922-OST0001 complete [10903.521168] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10903.531793] LustreError: Skipped 1 previous similar message [10903.542066] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10903.557884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10905.574126] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10905.579737] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10905.600090] Lustre: Skipped 1 previous similar message [10910.691892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10915.814068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10915.829183] Lustre: Skipped 2 previous similar messages [10916.319279] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10916.534534] Lustre: server umount lustre-OST0000 complete [10920.930523] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10920.954098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10920.963085] Lustre: Skipped 1 previous similar message [10931.170644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10931.178419] Lustre: Skipped 2 previous similar messages [10934.239143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10934.420073] Lustre: server umount lustre-MDT0000 complete [10937.640781] LustreError: 302158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789606993 with bad export cookie 10118654227101755412 [10937.643659] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10937.663798] LustreError: 302158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10937.943696] Lustre: server umount lustre-MDT0001 complete [10947.433400] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [10951.441734] Key type lgssc unregistered [10951.739032] LNet: 315441:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10951.753571] LNetError: 315441:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10951.769847] LNet: Removed LNI 192.168.206.155@tcp [10952.944144] Key type .llcrypt unregistered [10952.946101] Key type ._llcrypt unregistered [10979.438226] Key type ._llcrypt registered [10979.439590] Key type .llcrypt registered [10979.555183] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [10993.793713] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [10995.374109] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10995.499514] alg: No test for adler32 (adler32-zlib) [10996.562844] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [10996.879673] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [10998.567198] Key type lgssc registered [10999.933073] Lustre: Echo OBD driver; http://www.lustre.org/ [11034.588164] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11034.612504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11035.870849] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11035.898549] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11035.988450] Lustre: lustre-MDT0000: new disk, initializing [11036.067276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11036.124155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11040.532043] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11049.184288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11049.305805] Lustre: 319296:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11049.359396] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11049.366101] Lustre: Skipped 1 previous similar message [11049.450875] Lustre: lustre-MDT0001: new disk, initializing [11049.546254] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11049.584042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11049.590949] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11053.399325] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11062.177694] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11068.436368] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11075.633816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11075.876404] Lustre: lustre-OST0000: new disk, initializing [11075.884698] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11075.897919] Lustre: 321268:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11076.019389] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11082.019928] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11085.387768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11085.405853] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11085.439607] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11089.863116] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11091.587181] Lustre: Mounted lustre-client - version 2.17.58_39_g6aa67ea [11092.530776] Lustre: Unmounted lustre-client [11106.272247] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11106.274947] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11106.277274] LustreError: Skipped 1 previous similar message [11106.281033] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11106.290697] Lustre: Skipped 1 previous similar message [11106.307468] Lustre: Skipped 1 previous similar message [11107.297556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11107.307796] Lustre: Skipped 1 previous similar message [11108.740336] Lustre: server umount lustre-OST0000 complete [11112.419579] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11112.429340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11112.439322] Lustre: Skipped 1 previous similar message [11112.455557] Lustre: Skipped 1 previous similar message [11117.538263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11117.549919] Lustre: Skipped 1 previous similar message [11122.656275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11122.665785] Lustre: Skipped 1 previous similar message [11126.239678] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11126.447929] Lustre: server umount lustre-MDT0000 complete [11129.626788] LustreError: 320094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789607185 with bad export cookie 2890048492477338775 [11129.635852] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11129.652138] LustreError: 320094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11129.906333] Lustre: server umount lustre-MDT0001 complete [11137.859586] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [11141.300446] Key type lgssc unregistered [11152.927791] LNet: 323426:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11152.937424] LNetError: 323426:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11220.532602] LNet: Removed LNI 192.168.206.155@tcp [11221.511046] Key type .llcrypt unregistered [11221.513651] Key type ._llcrypt unregistered [11236.993871] Key type ._llcrypt registered [11236.995575] Key type .llcrypt registered [11237.101815] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [11252.432682] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [11253.229522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11253.248911] alg: No test for adler32 (adler32-zlib) [11254.406531] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [11254.629663] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [11256.287415] Key type lgssc registered [11257.276329] Lustre: Echo OBD driver; http://www.lustre.org/ [11282.952740] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 21:08:58 (1789607338) [11288.654399] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11288.677920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11289.902766] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11289.946173] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11290.044365] Lustre: lustre-MDT0000: new disk, initializing [11290.140412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11290.176277] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11294.034619] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11303.433184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11303.522240] Lustre: 326644:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11303.568978] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11303.573384] Lustre: Skipped 1 previous similar message [11303.657324] Lustre: lustre-MDT0001: new disk, initializing [11303.767782] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11303.812800] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11303.837794] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11307.324408] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11317.173150] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11323.244830] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11330.751730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11331.033266] Lustre: lustre-OST0000: new disk, initializing [11331.038488] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11331.050184] Lustre: 328614:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11331.194155] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11337.821859] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11340.878816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11340.892111] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11341.032176] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11346.580991] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11361.760826] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11361.766142] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11361.779772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11361.810350] Lustre: Skipped 1 previous similar message [11366.623316] Lustre: server umount lustre-OST0000 complete [11370.464224] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11370.468047] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11370.474363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11370.477870] Lustre: Skipped 1 previous similar message [11372.000594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11372.008178] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11375.085376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11375.090777] Lustre: Skipped 2 previous similar messages [11380.192485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11380.197168] Lustre: Skipped 1 previous similar message [11384.287170] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11384.442512] Lustre: server umount lustre-MDT0000 complete [11387.876494] LustreError: 326637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789607444 with bad export cookie 6191686928414385039 [11387.886445] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11387.890334] LustreError: 326637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11388.249068] Lustre: server umount lustre-MDT0001 complete [11396.801322] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [11399.054210] Key type lgssc unregistered [11399.313677] LNet: 330521:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11399.322452] LNetError: 330521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11399.339056] LNet: Removed LNI 192.168.206.155@tcp [11399.969917] Key type .llcrypt unregistered [11399.977093] Key type ._llcrypt unregistered [11418.909883] Key type ._llcrypt registered [11418.913634] Key type .llcrypt registered [11419.036727] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [11420.350525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11420.392430] alg: No test for adler32 (adler32-zlib) [11421.440576] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [11421.741374] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [11423.447267] Key type lgssc registered [11424.540535] Lustre: Echo OBD driver; http://www.lustre.org/ [11432.431830] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11432.461438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11433.846946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11437.937784] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11444.193775] LustreError: 331792:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11444.209880] LustreError: 331792:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11444.696891] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11444.970815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11448.208436] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11454.429652] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11459.623740] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11467.111156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11467.448021] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11475.008824] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11482.454466] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11491.119505] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11492.665800] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11496.417262] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11496.431877] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11496.458166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11499.625947] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.55@tcp (stopping) [11500.517695] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11500.530818] Lustre: Skipped 1 previous similar message [11502.593479] Lustre: server umount lustre-MDT0000 complete [11504.722456] LustreError: 331786:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11506.067252] LustreError: 331772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789607562 with bad export cookie 14240501480153036468 [11506.079610] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11506.084953] LustreError: 331772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11509.841438] LustreError: 331786:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11509.856303] LustreError: 331786:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11509.874317] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.55@tcp (stopping) [11509.884084] Lustre: Skipped 3 previous similar messages [11510.753322] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11512.592900] Lustre: server umount lustre-MDT0001 complete [11516.215420] Lustre: server umount lustre-OST0000 complete [11524.846550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11528.987538] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11531.584878] Lustre: 335968:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.155@tcp [11533.754811] Lustre: server umount MGS complete [11542.311023] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11542.861599] LustreError: 336541:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11542.875236] LustreError: 336541:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11542.910774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11547.931246] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11548.128705] LustreError: 336542:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11556.396652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11556.433030] LustreError: 336537:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11556.455894] LustreError: 336537:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11556.845026] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11561.084694] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11567.617155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11567.906923] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11572.825019] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11578.204604] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11579.429991] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11583.838617] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11585.175735] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11589.605941] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11589.611979] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11589.626814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11589.630434] Lustre: Skipped 2 previous similar messages [11592.682609] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11595.789660] Lustre: server umount lustre-OST0000 complete [11602.913769] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11602.915586] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11602.923417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11602.923426] Lustre: Skipped 2 previous similar messages [11602.928967] Lustre: Skipped 2 previous similar messages [11604.927983] Lustre: server umount lustre-MDT0000 complete [11607.698060] LustreError: 336522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789607663 with bad export cookie 14240501480153037308 [11607.701265] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11607.712785] LustreError: 336522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11607.946974] Lustre: server umount lustre-MDT0001 complete [11617.166720] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [11619.573753] Key type lgssc unregistered [11619.830821] LNet: 339977:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11619.837035] LNetError: 339977:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11619.856754] LNet: Removed LNI 192.168.206.155@tcp [11620.605904] Key type .llcrypt unregistered [11620.608438] Key type ._llcrypt unregistered [11637.564613] Key type ._llcrypt registered [11637.569904] Key type .llcrypt registered [11637.716245] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [11651.527893] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [11652.424126] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11652.545545] alg: No test for adler32 (adler32-zlib) [11653.534537] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [11653.805866] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [11655.439879] Key type lgssc registered [11656.405784] Lustre: Echo OBD driver; http://www.lustre.org/ [11688.794762] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11688.824582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11690.007591] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11690.048400] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11690.127103] Lustre: lustre-MDT0000: new disk, initializing [11690.182260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11690.191566] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11693.303996] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11701.481206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11701.549850] Lustre: 343135:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11701.591017] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [11701.591350] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11701.600693] Lustre: Skipped 1 previous similar message [11701.644756] Lustre: lustre-MDT0001: new disk, initializing [11701.736369] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11701.790696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11701.804970] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11706.067162] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11714.768493] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11720.375972] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11726.561382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11726.900549] Lustre: lustre-OST0000: new disk, initializing [11726.904375] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11726.910469] Lustre: 345102:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11727.006529] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11732.992747] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11735.120077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11735.145963] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11735.225497] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11740.779894] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11744.517771] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11744.744684] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11749.534363] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11749.749201] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11756.007683] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11756.011397] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11756.013227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11756.029923] Lustre: Skipped 1 previous similar message [11756.060506] Lustre: Skipped 1 previous similar message [11757.586992] Lustre: server umount lustre-OST0000 complete [11761.131032] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11761.150515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11761.152680] Lustre: Skipped 1 previous similar message [11766.240770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11767.220970] Lustre: server umount lustre-MDT0000 complete [11770.889030] LustreError: 343126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789607827 with bad export cookie 10564330360166109214 [11770.895405] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11770.899317] LustreError: 343126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11771.334706] Lustre: server umount lustre-MDT0001 complete [11779.174849] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 21:17:14 (1789607834) [11794.466208] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [11803.140788] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [11838.409146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11838.689227] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11838.731172] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11838.854990] Lustre: lustre-MDT0000: new disk, initializing [11838.933670] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11838.948451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11843.288285] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11852.877078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11853.078537] Lustre: 349858:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11853.164849] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11853.176589] Lustre: Skipped 1 previous similar message [11853.285357] Lustre: lustre-MDT0001: new disk, initializing [11853.405154] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11853.465211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11853.480556] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11858.036791] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11865.943030] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11870.444581] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11875.083247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11875.288751] Lustre: lustre-OST0000: new disk, initializing [11875.294749] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11875.308929] Lustre: 351828:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11875.406656] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11880.906174] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11884.667100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11884.678226] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11884.849870] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11890.297276] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11920.867225] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11920.870341] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11920.878350] LustreError: Skipped 2 previous similar messages [11920.887322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11920.889378] Lustre: Skipped 2 previous similar messages [11920.904873] Lustre: Skipped 2 previous similar messages [11922.734515] Lustre: server umount lustre-OST0000 complete [11930.080949] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11930.097417] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11930.128435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11932.477535] Lustre: server umount lustre-MDT0000 complete [11935.858481] LustreError: 349851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789607992 with bad export cookie 10564330360166110096 [11935.863508] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11935.883824] LustreError: 349851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11936.171274] Lustre: server umount lustre-MDT0001 complete [11946.449265] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [11949.330057] Key type lgssc unregistered [11949.711037] LNet: 353980:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11949.730444] LNetError: 353980:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11949.753869] LNet: Removed LNI 192.168.206.155@tcp [11950.854693] Key type .llcrypt unregistered [11950.857201] Key type ._llcrypt unregistered [12041.345943] Key type ._llcrypt registered [12041.349742] Key type .llcrypt registered [12041.474861] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [12041.747660] Key type .llcrypt unregistered [12041.749201] Key type ._llcrypt unregistered [12116.680857] Key type ._llcrypt registered [12116.684437] Key type .llcrypt registered [12116.835939] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [12117.331808] Key type .llcrypt unregistered [12117.338063] Key type ._llcrypt unregistered [12146.689242] Key type ._llcrypt registered [12146.690467] Key type .llcrypt registered [12146.752112] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [12147.066424] Key type .llcrypt unregistered [12147.072492] Key type ._llcrypt unregistered [12178.146359] Key type ._llcrypt registered [12178.148038] Key type .llcrypt registered [12178.217623] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [12178.408322] Key type .llcrypt unregistered [12178.411091] Key type ._llcrypt unregistered [12246.398968] Key type ._llcrypt registered [12246.401612] Key type .llcrypt registered [12246.492856] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [12246.701383] Key type .llcrypt unregistered [12246.703719] Key type ._llcrypt unregistered [12269.778707] Key type ._llcrypt registered [12269.781318] Key type .llcrypt registered [12269.881985] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [12270.149658] Key type .llcrypt unregistered [12270.155289] Key type ._llcrypt unregistered [12298.500385] Key type ._llcrypt registered [12298.506816] Key type .llcrypt registered [12298.616721] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [12298.878589] Key type .llcrypt unregistered [12298.881065] Key type ._llcrypt unregistered [12327.645594] Key type ._llcrypt registered [12327.647534] Key type .llcrypt registered [12327.789447] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [12328.028168] Key type .llcrypt unregistered [12328.033518] Key type ._llcrypt unregistered [12356.351241] Key type ._llcrypt registered [12356.355598] Key type .llcrypt registered [12356.462312] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [12356.803706] Key type .llcrypt unregistered [12356.813064] Key type ._llcrypt unregistered [12384.676053] Key type ._llcrypt registered [12384.678566] Key type .llcrypt registered [12384.792145] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [12385.104319] Key type .llcrypt unregistered [12385.107450] Key type ._llcrypt unregistered [12452.233352] Key type ._llcrypt registered [12452.239362] Key type .llcrypt registered [12452.327510] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [12452.728286] Key type .llcrypt unregistered [12452.735783] Key type ._llcrypt unregistered [12477.365909] Key type ._llcrypt registered [12477.368528] Key type .llcrypt registered [12477.456330] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [12477.712869] Key type .llcrypt unregistered [12477.717979] Key type ._llcrypt unregistered [12501.823091] Key type ._llcrypt registered [12501.826659] Key type .llcrypt registered [12501.932361] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [12502.171418] Key type .llcrypt unregistered [12502.175952] Key type ._llcrypt unregistered [12547.171413] Key type ._llcrypt registered [12547.176947] Key type .llcrypt registered [12547.338691] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [12560.089981] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [12560.878809] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12561.002360] alg: No test for adler32 (adler32-zlib) [12562.010964] Lustre: Lustre: Build Version: 2.17.58_39_g6aa67ea [12562.198292] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [12563.871269] Key type lgssc registered [12564.895570] Lustre: Echo OBD driver; http://www.lustre.org/ [12596.986782] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12597.014469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12598.458622] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [12598.513107] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [12598.663061] Lustre: tst250fs-MDT0000: new disk, initializing [12598.778542] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12598.800130] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12603.491539] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [12612.864640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12613.008733] Lustre: 374156:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12613.095167] Lustre: tst250fs-MDT0001: Not available for connect from 0@lo (not set up) [12613.099161] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [12613.110343] Lustre: Skipped 1 previous similar message [12613.198895] Lustre: tst250fs-MDT0001: new disk, initializing [12613.276683] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12613.321689] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12613.332244] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12616.799968] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [12620.770899] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12625.805708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12626.083696] Lustre: tst250fs-OST0000: new disk, initializing [12626.087750] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [12626.094310] Lustre: 375789:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12626.184169] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12629.753153] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12629.767801] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12629.863147] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12631.049151] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [12639.840632] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12639.991776] Lustre: tst250fs-OST0001: new disk, initializing [12639.998605] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [12640.003731] Lustre: 376660:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12640.079476] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12644.106144] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [12645.935307] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12645.948066] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12645.982343] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12648.429715] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [12666.342495] Lustre: tst250fs-MDT0000-lwp-OST0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12666.348217] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12666.361946] Lustre: Skipped 3 previous similar messages [12666.369263] Lustre: Skipped 2 previous similar messages [12671.460028] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12671.468985] Lustre: Skipped 4 previous similar messages [12671.626508] Lustre: server umount tst250fs-MDT0000 complete [12675.171293] LustreError: 374150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789608731 with bad export cookie 13177290489005438531 [12675.180509] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12675.183642] LustreError: 374150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12675.556344] Lustre: server umount tst250fs-MDT0001 complete [12688.073590] Lustre: server umount tst250fs-OST0000 complete [12701.819123] Lustre: server umount tst250fs-OST0001 complete [12717.601318] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [12723.789308] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [12746.575661] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 21:33:22 (1789608802) [12747.710696] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [12759.026148] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [12766.112685] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [12789.316818] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12495 sec ========= 21:34:05 (1789608845) [12790.496768] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 21:34:06 (1789608846) === [12793.101973] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 21:34:09 (1789608849) === [12824.634667] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [12827.072570] Key type lgssc unregistered [12827.382311] LNet: 385777:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12827.389945] LNetError: 385777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12827.416191] LNet: Removed LNI 192.168.206.155@tcp [12828.267154] Key type .llcrypt unregistered [12828.269164] Key type ._llcrypt unregistered