[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 531124852 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002338] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004012] kvm-guest: setup PV IPIs [ 0.006917] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007041] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008012] pid_max: default: 32768 minimum: 301 [ 0.009122] LSM: Security Framework initializing [ 0.010052] Yama: becoming mindful. [ 0.011024] SELinux: Initializing. [ 0.012008] *** VALIDATE selinux *** [ 0.018855] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023614] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.024159] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025134] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026122] *** VALIDATE tmpfs *** [ 0.028289] *** VALIDATE proc *** [ 0.029236] *** VALIDATE cgroup *** [ 0.030007] *** VALIDATE cgroup2 *** [ 0.031257] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.032144] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.033007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.034029] Spectre V2 : User space: Vulnerable [ 0.035006] Speculative Store Bypass: Vulnerable [ 0.038353] debug: unmapping init [mem 0xffffffffb5a59000-0xffffffffb5a60fff] [ 0.041000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.041693] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.042025] ... version: 2 [ 0.043013] ... bit width: 48 [ 0.044015] ... generic registers: 4 [ 0.045013] ... value mask: 0000ffffffffffff [ 0.046015] ... max period: 00007fffffffffff [ 0.047016] ... fixed-purpose events: 3 [ 0.048014] ... event mask: 000000070000000f [ 0.049317] rcu: Hierarchical SRCU implementation. [ 0.051506] smp: Bringing up secondary CPUs ... [ 0.052607] x86: Booting SMP configuration: [ 0.053031] .... node #0, CPUs: #1 #2 #3 [ 0.057588] smp: Brought up 1 node, 4 CPUs [ 0.059016] smpboot: Max logical packages: 1 [ 0.060023] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.141791] node 0 deferred pages initialised in 78ms [ 0.144026] devtmpfs: initialized [ 0.145271] x86/mm: Memory block size: 128MB [ 0.148919] gcov: version magic: 0x41383552 [ 0.151258] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.152082] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.153283] pinctrl core: initialized pinctrl subsystem [ 0.154176] [ 0.154738] ************************************************************* [ 0.155014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.156013] ** ** [ 0.157011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.158010] ** ** [ 0.159013] ** This means that this kernel is built to expose internal ** [ 0.160009] ** IOMMU data structures, which may compromise security on ** [ 0.161009] ** your system. ** [ 0.162009] ** ** [ 0.163007] ** If you see this message and you are not debugging the ** [ 0.164025] ** kernel, report this immediately to your vendor! ** [ 0.165007] ** ** [ 0.166010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.167010] ************************************************************* [ 0.168598] NET: Registered protocol family 16 [ 0.169289] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.170054] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.171037] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.172331] cpuidle: using governor menu [ 0.173247] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.176475] PCI: Using configuration type 1 for base access [ 0.178093] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.186059] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.187013] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.190032] cryptd: max_cpu_qlen set to 1000 [ 0.192298] ACPI: Added _OSI(Module Device) [ 0.193018] ACPI: Added _OSI(Processor Device) [ 0.194016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.195013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.199143] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.202571] ACPI: Interpreter enabled [ 0.203067] ACPI: PM: (supports S0 S3 S4 S5) [ 0.204012] ACPI: Using IOAPIC for interrupt routing [ 0.205128] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.206391] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.216532] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.217042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.218024] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.219088] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.221424] acpiphp: Slot [2] registered [ 0.222122] acpiphp: Slot [5] registered [ 0.223155] acpiphp: Slot [6] registered [ 0.224115] acpiphp: Slot [7] registered [ 0.225116] acpiphp: Slot [8] registered [ 0.226145] acpiphp: Slot [9] registered [ 0.227143] acpiphp: Slot [10] registered [ 0.228143] acpiphp: Slot [3] registered [ 0.229100] acpiphp: Slot [4] registered [ 0.230102] acpiphp: Slot [11] registered [ 0.231105] acpiphp: Slot [12] registered [ 0.232128] acpiphp: Slot [13] registered [ 0.233111] acpiphp: Slot [14] registered [ 0.234109] acpiphp: Slot [15] registered [ 0.235129] acpiphp: Slot [16] registered [ 0.236097] acpiphp: Slot [17] registered [ 0.237089] acpiphp: Slot [18] registered [ 0.238120] acpiphp: Slot [19] registered [ 0.239122] acpiphp: Slot [20] registered [ 0.240086] acpiphp: Slot [21] registered [ 0.240978] acpiphp: Slot [22] registered [ 0.241101] acpiphp: Slot [23] registered [ 0.242070] acpiphp: Slot [24] registered [ 0.242939] acpiphp: Slot [25] registered [ 0.243133] acpiphp: Slot [26] registered [ 0.244105] acpiphp: Slot [27] registered [ 0.245117] acpiphp: Slot [28] registered [ 0.246122] acpiphp: Slot [29] registered [ 0.247118] acpiphp: Slot [30] registered [ 0.248115] acpiphp: Slot [31] registered [ 0.249070] PCI host bridge to bus 0000:00 [ 0.250022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.251035] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.252038] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.254029] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.257027] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.259045] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.261324] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.265045] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.268398] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.277710] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.283117] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.284014] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.286013] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.287017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.289428] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.291506] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.292027] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.294618] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.300016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.315019] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.322014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.327103] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.338016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.342014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.356013] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.367674] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.378017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.385017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.405019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.414790] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.420021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.424017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.438019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.446071] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.451014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.456014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.467021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.474000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.477013] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.482014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.493015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.503409] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.508013] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.512012] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.523027] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.533579] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.537401] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.540388] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.542336] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.544160] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.547478] iommu: Default domain type: Passthrough [ 0.549528] SCSI subsystem initialized [ 0.551117] ACPI: bus type USB registered [ 0.552070] usbcore: registered new interface driver usbfs [ 0.553069] usbcore: registered new interface driver hub [ 0.555083] usbcore: registered new device driver usb [ 0.556176] pps_core: LinuxPPS API ver. 1 registered [ 0.559013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.561058] PTP clock support registered [ 0.563164] EDAC MC: Ver: 3.0.0 [ 0.565097] PCI: Using ACPI for IRQ routing [ 0.566748] NetLabel: Initializing [ 0.568007] NetLabel: domain hash size = 128 [ 0.568881] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.570083] NetLabel: unlabeled traffic allowed by default [ 0.571102] vgaarb: loaded [ 0.572224] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.573009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.579894] clocksource: Switched to clocksource kvm-clock [ 0.701485] VFS: Disk quotas dquot_6.6.0 [ 0.703177] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.705644] *** VALIDATE ramfs *** [ 0.706829] *** VALIDATE hugetlbfs *** [ 0.708343] pnp: PnP ACPI init [ 0.711613] pnp: PnP ACPI: found 6 devices [ 0.731042] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.734768] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.736966] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.739220] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.741888] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.744568] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.747714] NET: Registered protocol family 2 [ 0.750298] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.755137] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.758061] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.762611] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.766139] TCP: Hash tables configured (established 65536 bind 65536) [ 0.769138] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.772347] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.775446] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.778224] NET: Registered protocol family 1 [ 0.780615] RPC: Registered named UNIX socket transport module. [ 0.782728] RPC: Registered udp transport module. [ 0.784698] RPC: Registered tcp transport module. [ 0.786730] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.789501] NET: Registered protocol family 44 [ 0.791534] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.793801] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.796643] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.799236] PCI: CLS 0 bytes, default 64 [ 0.800648] Unpacking initramfs... [ 2.131613] debug: unmapping init [mem 0xffff92c03cc54000-0xffff92c03ffbffff] [ 2.135815] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.138098] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.140384] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.616443] Initialise system trusted keyrings [ 2.617489] Key type blacklist registered [ 2.618669] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.625691] zbud: loaded [ 2.628292] *** VALIDATE nfs *** [ 2.629100] *** VALIDATE nfs4 *** [ 2.630071] pstore: using deflate compression [ 2.632804] Platform Keyring initialized [ 2.729440] NET: Registered protocol family 38 [ 2.730784] Key type asymmetric registered [ 2.731658] Asymmetric key parser 'x509' registered [ 2.732842] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.734732] io scheduler mq-deadline registered [ 2.735672] io scheduler kyber registered [ 2.736676] io scheduler bfq registered [ 2.737748] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.739824] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.741805] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.743652] ACPI: Power Button [PWRF] [ 2.747436] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.753329] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.763037] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.768578] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.781176] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.807694] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.834156] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.838288] Non-volatile memory driver v1.3 [ 2.839308] Linux agpgart interface v0.103 [ 2.865697] virtio_blk virtio1: [vda] 149392 512-byte logical blocks (76.5 MB/72.9 MiB) [ 2.867529] vda: detected capacity change from 0 to 76488704 [ 2.881528] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.883592] vdb: detected capacity change from 0 to 1073741824 [ 2.895945] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.897862] vdc: detected capacity change from 0 to 2621440000 [ 2.915442] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.918360] vdd: detected capacity change from 0 to 2621440000 [ 2.940329] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.943368] vde: detected capacity change from 0 to 4294967296 [ 2.960288] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.963229] vdf: detected capacity change from 0 to 4294967296 [ 2.973192] libphy: Fixed MDIO Bus: probed [ 2.984608] usbcore: registered new interface driver usbserial_generic [ 2.987915] usbserial: USB Serial support registered for generic [ 2.990615] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.996254] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.998425] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.001204] mousedev: PS/2 mouse device common for all mice [ 3.004436] rtc_cmos 00:05: RTC can wake from S4 [ 3.007439] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.009726] rtc_cmos 00:05: registered as rtc0 [ 3.015675] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.016635] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.019115] intel_pstate: CPU model not supported [ 3.024542] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.028880] hid: raw HID events driver (C) Jiri Kosina [ 3.030705] usbcore: registered new interface driver usbhid [ 3.032507] usbhid: USB HID core driver [ 3.033915] drop_monitor: Initializing network drop monitor service [ 3.036078] Initializing XFRM netlink socket [ 3.038270] NET: Registered protocol family 10 [ 3.041374] Segment Routing with IPv6 [ 3.042709] NET: Registered protocol family 17 [ 3.045088] mpls_gso: MPLS GSO support [ 3.051557] RAS: Correctable Errors collector initialized. [ 3.053917] AVX version of gcm_enc/dec engaged. [ 3.055817] AES CTR mode by8 optimization enabled [ 3.123579] sched_clock: Marking stable (3123560555, 0)->(4054920389, -931359834) [ 3.126298] registered taskstats version 1 [ 3.127769] Loading compiled-in X.509 certificates [ 3.129554] zswap: loaded using pool lzo/zbud [ 3.149108] Key type big_key registered [ 3.161342] Key type encrypted registered [ 3.163299] ima: No TPM chip found, activating TPM-bypass! [ 3.165813] ima: Allocated hash algorithm: sha1 [ 3.167712] ima: No architecture policies found [ 3.168936] evm: Initialising EVM extended attributes: [ 3.169997] evm: security.selinux [ 3.170771] evm: security.ima [ 3.171454] evm: security.capability [ 3.172391] evm: HMAC attrs: 0x1 [ 3.174514] rtc_cmos 00:05: setting system clock to 2026-08-22 10:37:25 UTC (1787395045) [ 3.181182] debug: unmapping init [mem 0xffffffffb6a03000-0xffffffffb6bfffff] [ 3.183762] debug: unmapping init [mem 0xffffffffb5782000-0xffffffffb5a58fff] [ 3.191058] Write protecting the kernel read-only data: 28672k [ 3.193198] debug: unmapping init [mem 0xffffffffb3e03000-0xffffffffb3ffffff] [ 3.194921] debug: unmapping init [mem 0xffffffffb4714000-0xffffffffb47fffff] [ 3.228583] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.235623] systemd[1]: Detected virtualization kvm. [ 3.237283] systemd[1]: Detected architecture x86-64. [ 3.238464] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.265684] systemd[1]: No hostname configured. [ 3.267055] systemd[1]: Set hostname to . [ 3.268482] random: systemd: uninitialized urandom read (16 bytes read) [ 3.270111] systemd[1]: Initializing machine ID from random generator. [ 3.393912] random: systemd: uninitialized urandom read (16 bytes read) [ 3.396634] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.400767] random: systemd: uninitialized urandom read (16 bytes read) [ 3.402516] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.405751] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. [ OK ] Reached target Timers. Starting Journal Service... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.979082] device-mapper: uevent: version 1.0.3 [ 3.981624] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization.[ 4.584821] random: fast init done [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.649453] virtio_net virtio0 ens2: renamed from eth0 [ 4.652278] scsi host0: ata_piix [ 4.674489] scsi host1: ata_piix [ 4.676025] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.678207] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.324226] dracut-initqueue[583]: RTNETLINK answers: File exists [ 9.681673] random: crng init done [ 9.683178] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 9.934683] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 10.959748] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.194720] SELinux: Disabled at runtime. [ 11.256874] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.269665] systemd[1]: Detected virtualization kvm. [ 11.272458] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.661828] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.664426] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.668519] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.672138] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.676229] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.682146] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.685674] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Remount Root and Kernel File Systems... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... [ 11.728251] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... Starting udev Coldplug all Devices... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.033504] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.251367] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.362742] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.460885] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.477286] EDAC sbridge: Ver: 1.1.2 [ 14.299794] Key type dns_resolver registered [ 14.628812] NFS: Registering the id_resolver key type [ 14.630932] Key type id_resolver registered [ 14.632301] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg156-server login: [ 57.782901] hrtimer: interrupt took 10016598 ns [ 68.951657] libcfs: loading out-of-tree module taints kernel. [ 69.033985] Key type ._llcrypt registered [ 69.038146] Key type .llcrypt registered [ 69.181397] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 90.114261] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 92.164696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 92.189873] alg: No test for adler32 (adler32-zlib) [ 93.777472] Lustre: Lustre: Build Version: 2.17.54_108_g575fb00 [ 94.931177] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 96.703180] Key type lgssc registered [ 98.601171] Lustre: Echo OBD driver; http://www.lustre.org/ [ 118.229834] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 167.473380] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 181.496171] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 181.540039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 182.805480] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 182.832170] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 182.918918] Lustre: lustre-MDT0000: new disk, initializing [ 183.046402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 183.067251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 187.546651] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 201.598692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 201.715614] Lustre: 6526:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 201.771585] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 201.777158] Lustre: Skipped 1 previous similar message [ 201.925521] Lustre: lustre-MDT0001: new disk, initializing [ 202.000394] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 202.032557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 202.048101] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 207.000217] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 212.366413] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 222.503908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 222.871448] Lustre: lustre-OST0000: new disk, initializing [ 222.884571] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 222.892624] Lustre: 8464:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 222.991402] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 230.590571] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 231.021244] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 231.046133] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 231.105475] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 245.663600] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 245.763176] Lustre: lustre-OST0001: new disk, initializing [ 245.766859] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 245.771377] Lustre: 9535:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 245.826627] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 252.372974] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 253.522810] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 253.538601] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 253.623040] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 264.963824] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 274.737772] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 283.072865] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing check_logdir /tmp/testlogs/ [ 290.014875] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing yml_node [ 294.978805] Lustre: DEBUG MARKER: Client: 2.17.54.108 [ 297.819425] Lustre: DEBUG MARKER: MDS: 2.17.54.108 [ 301.026562] Lustre: DEBUG MARKER: OSS: 2.17.54.108 [ 303.593442] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 22 06:42:23 EDT 2026 [ 323.133427] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 324.939649] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 345.568197] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 345.591745] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 345.645780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 347.793419] Lustre: server umount lustre-MDT0000 complete [ 352.451935] LustreError: 6516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395394 with bad export cookie 3731685739780046602 [ 352.465932] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 352.473381] LustreError: 6516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 356.320526] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 356.322790] LustreError: 6533:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 356.349233] Lustre: Skipped 3 previous similar messages [ 356.354848] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 356.378896] LustreError: 6533:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 356.401060] Lustre: Skipped 5 previous similar messages [ 357.367304] LustreError: 8263:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 357.368055] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 357.390367] LustreError: 8263:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 357.412776] Lustre: Skipped 1 previous similar message [ 358.984381] Lustre: server umount lustre-MDT0001 complete [ 363.534565] Lustre: server umount lustre-OST0000 complete [ 367.127061] Lustre: server umount lustre-OST0001 complete [ 373.162497] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 384.578197] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 418.260232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 418.511556] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 418.551126] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 418.607534] Lustre: lustre-MDT0000: new disk, initializing [ 418.665835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 418.677643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 423.062884] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 433.996034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 434.123081] Lustre: 15100:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 434.174535] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 434.178815] Lustre: Skipped 1 previous similar message [ 434.248795] Lustre: lustre-MDT0001: new disk, initializing [ 434.328948] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 434.380944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 434.396465] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 439.250644] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 449.471957] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 456.243504] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 462.810895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 463.101706] Lustre: lustre-OST0000: new disk, initializing [ 463.105796] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 463.112172] Lustre: 17069:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 463.223540] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 464.713546] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 464.720099] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 464.872149] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 470.341196] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 480.243492] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 485.020851] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 485.182925] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 490.046807] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 490.338274] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 495.585445] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 495.603022] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 495.610678] Lustre: Skipped 2 previous similar messages [ 495.630591] Lustre: Skipped 1 previous similar message [ 500.709459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 507.359185] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 507.700550] Lustre: server umount lustre-OST0000 complete [ 516.065195] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 516.066352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 516.071203] Lustre: Skipped 1 previous similar message [ 516.080099] Lustre: Skipped 4 previous similar messages [ 517.910667] Lustre: server umount lustre-MDT0000 complete [ 521.204315] LustreError: 15111:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 521.232870] LustreError: 15111:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 521.351751] LustreError: 15091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395563 with bad export cookie 3731685739780048135 [ 521.362677] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 521.365983] LustreError: 15091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 521.819270] Lustre: server umount lustre-MDT0001 complete [ 527.389770] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 06:46:08 (1787395568) [ 546.907681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 547.455851] LustreError: 19943:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 547.539214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 552.937787] LustreError: 19944:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 552.982952] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 562.155164] LustreError: 19943:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 562.181067] LustreError: 19943:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 562.981309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 569.113762] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 576.753662] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 582.349208] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 584.099882] Lustre: Failing over lustre-MDT0000 [ 584.160103] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 584.174361] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 584.198031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 584.396695] Lustre: server umount lustre-MDT0000 complete [ 588.256699] LustreError: 19944:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 588.280825] LustreError: 19944:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 603.038953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 603.167628] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 603.558366] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 607.912955] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 608.738172] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 608.750772] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 608.779498] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 617.380621] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 623.078124] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 623.098516] Lustre: Skipped 1 previous similar message [ 623.111429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 623.125421] Lustre: Skipped 1 previous similar message [ 636.385416] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 637.094243] Lustre: server umount lustre-MDT0000 complete [ 640.487105] LustreError: 19940:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 640.517249] LustreError: 19940:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 642.738748] LustreError: 19924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395685 with bad export cookie 3731685739780049549 [ 642.750670] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 642.754135] LustreError: 19924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 643.072236] Lustre: server umount lustre-MDT0001 complete [ 653.114586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 653.949737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 653.956218] Lustre: Skipped 2 previous similar messages [ 659.018395] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 667.648213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 673.067065] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 680.479552] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 686.300454] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 688.407568] Lustre: Failing over lustre-MDT0000 [ 688.423102] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 688.429659] Lustre: Skipped 1 previous similar message [ 688.433194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 688.437460] Lustre: Skipped 7 previous similar messages [ 688.654974] Lustre: server umount lustre-MDT0000 complete [ 707.665947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 707.787991] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 708.031431] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 712.415386] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 713.058388] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 713.062852] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 713.088352] Lustre: Skipped 1 previous similar message [ 713.099656] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 721.485281] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 728.547689] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 728.554140] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 728.569980] Lustre: Skipped 2 previous similar messages [ 731.539854] Lustre: server umount lustre-MDT0000 complete [ 733.664068] LustreError: 23282:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 733.678459] LustreError: 23282:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 734.859611] LustreError: 25942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395777 with bad export cookie 3731685739780050487 [ 734.864186] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 734.879166] LustreError: 25942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 735.221475] Lustre: server umount lustre-MDT0001 complete [ 742.747516] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 06:49:43 (1787395783) [ 756.947139] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 764.503459] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 798.606988] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 807.572479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 807.776666] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 807.799342] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 807.858799] Lustre: lustre-MDT0000: new disk, initializing [ 807.919524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 807.923191] Lustre: Skipped 2 previous similar messages [ 807.938192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 811.878585] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 820.711347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 820.782805] Lustre: 29652:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 820.911948] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 824.231287] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 831.993271] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 837.686950] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 845.692823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 846.139742] Lustre: 31620:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 848.156250] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 848.216788] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 848.264644] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 848.290482] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 854.173907] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 862.978994] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 868.369316] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 889.311995] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 889.331132] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 889.342852] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 889.342861] Lustre: Skipped 2 previous similar messages [ 889.387948] Lustre: Skipped 1 previous similar message [ 893.689874] Lustre: server umount lustre-OST0000 complete [ 898.020442] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 898.032073] LustreError: Skipped 1 previous similar message [ 904.217883] Lustre: server umount lustre-MDT0000 complete [ 904.685631] LustreError: 29663:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 904.704040] LustreError: 29663:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 908.696899] LustreError: 29645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395951 with bad export cookie 3731685739780050893 [ 908.720014] LustreError: 29645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 908.721124] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 909.462943] Lustre: server umount lustre-MDT0001 complete [ 924.776089] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 927.921774] Key type lgssc unregistered [ 928.205863] LNet: 33599:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 928.212724] LNetError: 33599:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 928.236717] LNet: Removed LNI 192.168.201.156@tcp [ 929.112250] Key type .llcrypt unregistered [ 929.118094] Key type ._llcrypt unregistered [ 953.913600] Key type ._llcrypt registered [ 953.916748] Key type .llcrypt registered [ 954.051435] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 954.904072] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 954.981366] alg: No test for adler32 (adler32-zlib) [ 956.024514] Lustre: Lustre: Build Version: 2.17.54_108_g575fb00 [ 956.313380] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 958.015208] Key type lgssc registered [ 959.378718] Lustre: Echo OBD driver; http://www.lustre.org/ [ 970.240464] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 970.280863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 971.921648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 978.366546] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 982.512181] LustreError: 35014:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 982.539100] LustreError: 35014:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 987.622627] LustreError: 35015:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 988.312473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 988.531631] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 992.715412] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1000.047871] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1005.736967] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1012.216884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1012.589516] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1013.633082] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1016.744694] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1018.678767] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1025.877714] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1038.799994] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 06:54:39 (1787396079) [ 1040.892540] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1042.959668] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1050.360873] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 06:54:51 (1787396091) [ 1059.557271] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 06:55:00 (1787396100) [ 1060.570118] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1069.075505] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1069.081632] Lustre: Skipped 2 previous similar messages [ 1070.713720] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 06:55:11 (1787396111) [ 1074.129370] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1074.134677] Lustre: Skipped 3 previous similar messages [ 1093.292517] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1093.308465] Lustre: Skipped 6 previous similar messages [ 1095.786586] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 06:55:36 (1787396136) [ 1098.675948] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1102.170519] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1111.166408] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 06:55:51 (1787396151) [ 1112.977395] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1112.981491] Lustre: Skipped 3 previous similar messages [ 1128.007103] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 06:56:08 (1787396168) [ 1144.801354] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1144.804277] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1144.815640] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1144.845233] Lustre: Skipped 1 previous similar message [ 1149.202755] Lustre: server umount lustre-OST0000 complete [ 1155.041812] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1155.072713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1155.085805] Lustre: Skipped 2 previous similar messages [ 1159.568185] Lustre: server umount lustre-MDT0000 complete [ 1160.164634] LustreError: 35014:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1160.174959] LustreError: 35014:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1163.572645] LustreError: 38934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396205 with bad export cookie 11518839753341917202 [ 1163.583723] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1163.599632] LustreError: 38934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1163.923076] Lustre: server umount lustre-MDT0001 complete [ 1174.772627] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 1177.112782] Key type lgssc unregistered [ 1177.393840] LNet: 42876:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1177.401957] LNetError: 42876:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1177.423233] LNet: Removed LNI 192.168.201.156@tcp [ 1178.338696] Key type .llcrypt unregistered [ 1178.340517] Key type ._llcrypt unregistered [ 1196.948710] Key type ._llcrypt registered [ 1196.950926] Key type .llcrypt registered [ 1197.116483] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 1213.150497] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 1214.266235] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1214.331083] alg: No test for adler32 (adler32-zlib) [ 1215.378638] Lustre: Lustre: Build Version: 2.17.54_108_g575fb00 [ 1215.637970] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 1217.319268] Key type lgssc registered [ 1218.521491] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1254.830105] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1254.853809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1256.140245] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1256.167587] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1256.258937] Lustre: lustre-MDT0000: new disk, initializing [ 1256.438367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1256.474515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1261.800790] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1272.385376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1272.511496] Lustre: 46028:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1272.568148] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1272.571583] Lustre: Skipped 1 previous similar message [ 1272.633875] Lustre: lustre-MDT0001: new disk, initializing [ 1272.705082] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1272.729091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1272.737688] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1276.706378] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1285.853977] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1291.475350] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1298.959189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1299.262349] Lustre: lustre-OST0000: new disk, initializing [ 1299.266644] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1299.275906] Lustre: 47996:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1299.399798] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1304.600467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1304.604990] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1304.681371] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1305.315418] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1316.248991] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1323.706453] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1324.143925] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1331.641987] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1332.052826] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1335.268388] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1335.285599] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1335.313212] Lustre: Skipped 1 previous similar message [ 1335.316287] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1335.330765] Lustre: Skipped 1 previous similar message [ 1340.393737] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1340.405071] Lustre: Skipped 1 previous similar message [ 1340.610524] Lustre: server umount lustre-OST0000 complete [ 1345.507566] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1345.530774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1345.534939] Lustre: Skipped 1 previous similar message [ 1345.545561] Lustre: Skipped 1 previous similar message [ 1350.624561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1350.634790] Lustre: Skipped 1 previous similar message [ 1355.746185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1355.767050] Lustre: Skipped 1 previous similar message [ 1359.327120] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1359.520722] Lustre: server umount lustre-MDT0000 complete [ 1363.212449] LustreError: 48008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396405 with bad export cookie 2255294096148002808 [ 1363.218469] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1363.234829] LustreError: 48008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1363.624212] Lustre: server umount lustre-MDT0001 complete [ 1374.034681] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 07:00:14 (1787396414) [ 1381.091589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1381.563518] LustreError: 50465:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1381.586801] LustreError: 50465:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1381.652063] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1386.038327] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1386.978168] LustreError: 50466:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1391.080180] LustreError: 50465:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1395.596938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1396.060548] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1401.460805] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1413.007054] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1420.890779] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1429.655369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1430.374963] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1438.694787] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1447.896659] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1458.390145] LustreError: 53009:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1459.807091] Lustre: Modifying parameter general.timeout=1 in log params [ 1460.885906] Lustre: Modifying parameter general.timeout=2 in log params [ 1463.246067] Lustre: Modifying parameter general.timeout=4 in log params [ 1463.254559] Lustre: Skipped 1 previous similar message [ 1467.286574] Lustre: Modifying parameter general.timeout=7 in log params [ 1467.314894] Lustre: Skipped 2 previous similar messages [ 1475.994328] Lustre: Modifying parameter general.timeout=16 in log params [ 1476.004660] Lustre: Skipped 8 previous similar messages [ 1492.497463] Lustre: Modifying parameter general.timeout=34 in log params [ 1492.505247] Lustre: Skipped 17 previous similar messages [ 1524.857483] Lustre: Modifying parameter general.timeout=69 in log params [ 1524.860395] Lustre: Skipped 34 previous similar messages [ 1592.805406] Lustre: Disabling parameter general.timeout= in log params [ 1592.808604] Lustre: Skipped 60 previous similar messages [ 1596.730667] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 07:03:56 (1787396636) [ 1600.095807] Lustre: 59954:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1601.885658] Lustre: 60062:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1603.963607] Lustre: 60162:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1608.171605] Lustre: 60371:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1608.179676] Lustre: 60371:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1641.012613] Lustre: 61834:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1641.021472] Lustre: 61834:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1641.042635] LustreError: 61834:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1646.775205] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 07:04:46 (1787396686) [ 1662.435258] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1662.450137] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1662.478282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1665.509913] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1665.525339] Lustre: Skipped 1 previous similar message [ 1668.745986] Lustre: server umount lustre-MDT0000 complete [ 1670.625922] LustreError: 50461:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1670.652749] LustreError: 50461:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1672.463209] LustreError: 50446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396714 with bad export cookie 2255294096148003690 [ 1672.467325] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1672.473658] LustreError: 50446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1672.987431] Lustre: server umount lustre-MDT0001 complete [ 1687.516383] Lustre: server umount lustre-OST0000 complete [ 1717.673831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1718.096657] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1718.554165] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1724.256229] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1734.749138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1734.879622] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1734.943271] Lustre: 64368:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1740.443884] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1748.362569] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1754.520929] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1762.174735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1762.428807] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1762.884778] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1762.887511] Lustre: Skipped 1 previous similar message [ 1769.803926] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1778.171888] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1780.230825] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1780.240420] Lustre: Skipped 23 previous similar messages [ 1780.302947] Lustre: 66934:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1789.922851] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1789.948656] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1789.966985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1789.974068] Lustre: Skipped 3 previous similar messages [ 1793.468385] Lustre: server umount lustre-OST0000 complete [ 1800.677844] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1800.694856] Lustre: Skipped 2 previous similar messages [ 1804.232549] Lustre: server umount lustre-MDT0000 complete [ 1805.792335] LustreError: 64380:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1805.810762] LustreError: 64380:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1807.845874] LustreError: 64360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396850 with bad export cookie 2255294096148006847 [ 1807.850971] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1807.864881] LustreError: 64360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1808.172189] Lustre: server umount lustre-MDT0001 complete [ 1821.105941] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 1823.888500] Key type lgssc unregistered [ 1824.241543] LNet: 68143:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1824.258208] LNetError: 68143:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1824.278261] LNet: Removed LNI 192.168.201.156@tcp [ 1825.051672] Key type .llcrypt unregistered [ 1825.054390] Key type ._llcrypt unregistered [ 1851.138678] Key type ._llcrypt registered [ 1851.141400] Key type .llcrypt registered [ 1851.236590] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 1852.912644] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1852.965072] alg: No test for adler32 (adler32-zlib) [ 1854.023971] Lustre: Lustre: Build Version: 2.17.54_108_g575fb00 [ 1854.400679] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 1856.095195] Key type lgssc registered [ 1857.352567] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1866.924255] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1866.949581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1868.586089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1875.141600] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1877.988360] LustreError: 69560:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1877.998330] LustreError: 69560:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1883.105565] LustreError: 69561:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1885.481778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1886.455467] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1891.619097] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1900.714622] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1907.239462] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1914.554833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1915.480490] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1923.090960] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1932.974545] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1946.700874] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1957.857671] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1957.864238] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1957.874672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1958.886592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1958.886779] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1958.930546] Lustre: Skipped 1 previous similar message [ 1962.381477] Lustre: server umount lustre-MDT0000 complete [ 1964.002166] LustreError: 69561:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1964.027880] LustreError: 69561:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1967.015847] LustreError: 69560:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1967.042510] LustreError: 69560:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1967.192484] LustreError: 69541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397009 with bad export cookie 5710718843238067958 [ 1967.198452] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1967.217878] LustreError: 69541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1967.932282] Lustre: server umount lustre-MDT0001 complete [ 1982.894731] Lustre: server umount lustre-OST0000 complete [ 2021.489646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2021.924754] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2021.984996] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2022.346800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2027.256472] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2036.838451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2036.964765] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2036.989239] Lustre: 74436:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2037.554783] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2043.143394] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2052.787377] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2060.611306] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2069.522340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2069.782912] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2070.319819] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2078.185277] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2087.462425] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2089.492541] Lustre: 77012:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2091.396700] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2099.172639] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2099.179333] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2099.191948] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2099.191956] Lustre: Skipped 2 previous similar messages [ 2099.244998] Lustre: Skipped 1 previous similar message [ 2103.120358] Lustre: server umount lustre-OST0000 complete [ 2108.896490] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2108.913215] LustreError: Skipped 1 previous similar message [ 2108.915185] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2108.929038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2108.931609] Lustre: Skipped 1 previous similar message [ 2113.955488] Lustre: server umount lustre-MDT0000 complete [ 2114.535086] LustreError: 74446:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2114.555649] LustreError: 74446:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2117.934090] LustreError: 74428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397160 with bad export cookie 5710718843238068805 [ 2117.948781] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2117.956801] LustreError: 74428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2118.353714] Lustre: server umount lustre-MDT0001 complete [ 2131.890550] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 2135.057623] Key type lgssc unregistered [ 2135.362700] LNet: 78223:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2135.369694] LNetError: 78223:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2135.382074] LNet: Removed LNI 192.168.201.156@tcp [ 2136.463563] Key type .llcrypt unregistered [ 2136.467421] Key type ._llcrypt unregistered [ 2163.572378] Key type ._llcrypt registered [ 2163.574643] Key type .llcrypt registered [ 2163.696471] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 2164.714114] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2164.815138] alg: No test for adler32 (adler32-zlib) [ 2165.875693] Lustre: Lustre: Build Version: 2.17.54_108_g575fb00 [ 2166.268107] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 2168.033334] Key type lgssc registered [ 2169.191193] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2177.913593] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2177.951861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2179.610132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2185.101221] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2188.772770] LustreError: 79636:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2188.784388] LustreError: 79636:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2193.889034] LustreError: 79635:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2194.152283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2194.750393] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2199.934993] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2208.099840] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2214.186329] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2221.491127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2222.133232] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2229.992876] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2239.848864] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2246.305202] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2247.585187] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2248.947025] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2251.511644] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2251.516561] Lustre: Skipped 1 previous similar message [ 2256.297696] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2256.302553] Lustre: Skipped 4 previous similar messages [ 2265.254306] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2265.279131] Lustre: Skipped 7 previous similar messages [ 2282.001147] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2282.013489] Lustre: Skipped 15 previous similar messages [ 2314.493621] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2314.500802] Lustre: Skipped 32 previous similar messages [ 2362.175791] Lustre: 87739:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2366.292369] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 07:16:46 (1787397406) [ 2368.599746] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2370.523840] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 07:16:51 (1787397411) [ 2377.699622] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2377.709986] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2377.713763] LustreError: Skipped 1 previous similar message [ 2377.715401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2377.726551] Lustre: Skipped 1 previous similar message [ 2377.735861] Lustre: Skipped 1 previous similar message [ 2382.826940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2382.830473] Lustre: Skipped 1 previous similar message [ 2383.141560] Lustre: server umount lustre-OST0000 complete [ 2387.937716] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2387.955307] Lustre: Skipped 1 previous similar message [ 2387.957500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2387.973946] Lustre: Skipped 1 previous similar message [ 2393.057311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2393.068319] Lustre: Skipped 1 previous similar message [ 2398.176452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2398.179446] Lustre: Skipped 1 previous similar message [ 2401.247140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2401.479930] Lustre: server umount lustre-MDT0000 complete [ 2403.299168] LustreError: 79635:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2403.322762] LustreError: 79635:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2405.418072] LustreError: 83610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397447 with bad export cookie 9962754573456544504 [ 2405.427099] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2405.434432] LustreError: 83610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2405.730349] Lustre: server umount lustre-MDT0001 complete [ 2415.992334] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 2418.790746] Key type lgssc unregistered [ 2419.120528] LNet: 88989:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2419.124867] LNetError: 88989:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2419.159272] LNet: Removed LNI 192.168.201.156@tcp [ 2420.304126] Key type .llcrypt unregistered [ 2420.305937] Key type ._llcrypt unregistered [ 2425.643762] Key type ._llcrypt registered [ 2425.651576] Key type .llcrypt registered [ 2425.835676] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2431.544967] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules [ 2432.952618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2432.988773] alg: No test for adler32 (adler32-zlib) [ 2434.051233] Lustre: Lustre: Build Version: 2.17.54_108_g575fb00 [ 2434.412686] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 2436.127864] Key type lgssc registered [ 2437.762609] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2439.247245] LustreError: 90201:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2460.218215] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 2460.817290] LustreError: 90201:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2470.251129] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2470.270371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2471.763983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2477.036743] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2481.121336] LustreError: 91218:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2481.134769] LustreError: 91218:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2486.242867] LustreError: 91217:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2486.790651] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 07:18:47 (1787397527) [ 2491.365605] LustreError: 91218:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2491.969568] LustreError: 91249:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2491.976750] LustreError: 91249:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2491.985475] LustreError: 91249:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 2492.300133] Lustre: server umount lustre-MDT0000 complete [ 2507.952278] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 2511.085825] Key type lgssc unregistered [ 2511.418887] LNet: 92743:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2511.428371] LNetError: 92743:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2511.444979] LNet: Removed LNI 192.168.201.156@tcp [ 2512.404561] Key type .llcrypt unregistered [ 2512.407203] Key type ._llcrypt unregistered [ 2534.883099] Key type ._llcrypt registered [ 2534.884759] Key type .llcrypt registered [ 2535.023095] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 2536.367899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2536.506666] alg: No test for adler32 (adler32-zlib) [ 2537.670407] Lustre: Lustre: Build Version: 2.17.54_108_g575fb00 [ 2538.044674] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 2539.775874] Key type lgssc registered [ 2541.099522] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2549.634400] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2549.670442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2551.279225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2556.002366] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2561.521379] LustreError: 93958:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2561.547799] LustreError: 93958:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2565.866676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2566.609390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2571.799336] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2580.710089] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2588.810265] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2596.736860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2597.655366] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2603.159168] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2611.231053] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2674.752911] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 07:21:55 (1787397715) [ 2676.835638] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2678.575619] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 07:21:59 (1787397719) [ 2683.873600] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2683.894320] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2683.910738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2687.972589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2687.975263] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2687.991186] Lustre: Skipped 1 previous similar message [ 2688.020490] Lustre: Skipped 1 previous similar message [ 2689.295824] Lustre: server umount lustre-MDT0000 complete [ 2693.088967] LustreError: 93959:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2693.119427] LustreError: 93959:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2693.686326] LustreError: 93941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397736 with bad export cookie 4981233687544459962 [ 2693.689337] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2693.698308] LustreError: 93941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2694.253277] Lustre: server umount lustre-MDT0001 complete [ 2708.228463] Lustre: server umount lustre-OST0000 complete [ 2720.242693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2720.831739] LustreError: 98283:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2721.074932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2726.020657] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2726.369610] LustreError: 98284:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2726.378179] LustreError: 98284:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2731.501555] LustreError: 98283:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2735.249130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2736.216067] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2741.772237] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2749.029743] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2754.523855] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2765.250810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2765.544747] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2765.555158] LustreError: 98272:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2765.586881] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2765.593903] LustreError: 100367:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2765.600874] LustreError: 100367:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2765.606897] LustreError: 100367:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2765.699440] Lustre: server umount lustre-OST0000 complete [ 2765.701140] LustreError: 100367:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2774.319867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2774.523413] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2774.534594] LustreError: 98270:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2774.566068] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2774.579167] LustreError: 100835:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2774.587257] LustreError: 100835:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2774.593030] LustreError: 100835:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2774.655635] Lustre: server umount lustre-OST0000 complete [ 2774.659267] LustreError: 100835:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2786.143600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2786.388754] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2786.397186] Lustre: Found index 0 for lustre-OST0000, updating log [ 2786.402818] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2786.490844] Lustre: lustre-OST0000: new disk, initializing [ 2786.494882] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2786.499962] Lustre: 101369:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2786.821092] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2792.417135] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2796.867634] Lustre: Failing over lustre-OST0000 [ 2797.089226] Lustre: server umount lustre-OST0000 complete [ 2801.128422] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2801.139239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2801.140278] Lustre: Skipped 1 previous similar message [ 2801.155906] Lustre: Skipped 2 previous similar messages [ 2806.239985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2806.254068] Lustre: Skipped 1 previous similar message [ 2811.450214] Lustre: server umount lustre-MDT0000 complete [ 2812.384620] LustreError: 99671:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2812.411798] LustreError: 99671:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2815.775157] LustreError: 100387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397858 with bad export cookie 4981233687544462706 [ 2815.782946] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2815.787657] LustreError: 100387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 2816.126518] Lustre: server umount lustre-MDT0001 complete [ 2825.748281] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 07:24:26 (1787397866) [ 2844.231190] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 2856.974252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2857.543256] LustreError: 103716:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2857.660552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2863.157407] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2873.297786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2878.901622] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2882.836642] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2890.445949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2891.230564] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2891.243970] Lustre: Skipped 1 previous similar message [ 2891.256685] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2898.152677] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2898.409392] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2898.423332] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 2898.456452] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2902.524234] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:05 [ 2902.541548] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2902.562635] Lustre: lustre-OST0000: Recovery over after 0:04, of 1 clients 1 recovered and 0 were evicted. [ 2902.565335] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2902.581678] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2908.846794] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2908.978443] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2909.139950] Lustre: lustre-OST0001: new disk, initializing [ 2909.144913] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2909.156309] Lustre: 106035:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2915.966302] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2916.419660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2916.430803] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2916.565906] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2925.065894] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2932.759723] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2939.409198] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.56@tcp (stopping) [ 2939.418329] Lustre: Skipped 2 previous similar messages [ 2940.386070] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2940.393198] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2943.929217] Lustre: server umount lustre-MDT0001 complete [ 2944.521251] LustreError: 105487:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2944.537284] LustreError: 105487:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2954.403355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2954.564018] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2954.575944] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2954.662588] Lustre: 103706:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2955.290412] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2955.293511] Lustre: Skipped 1 previous similar message [ 2956.343759] LustreError: 107609:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2956.349595] LustreError: 107609:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2956.355095] LustreError: 103708:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2956.427939] LustreError: 107620:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.156@tcp: cfg command failed: rc = -17 [ 2956.438049] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2956.438049] [ 2960.385938] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 2960.702638] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2970.795422] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 07:26:51 (1787398011) [ 2988.982169] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 3006.835679] Lustre: 109649:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3028.447948] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3032.216495] Lustre: 110784:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3119.071716] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3119.073303] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3119.073915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3119.073920] Lustre: Skipped 5 previous similar messages [ 3119.095587] Lustre: Skipped 5 previous similar messages [ 3121.650392] Lustre: server umount lustre-MDT0000 complete [ 3124.193516] LustreError: 103716:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3124.214646] LustreError: 103716:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 3125.182639] LustreError: 103698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787398167 with bad export cookie 4981233687544463707 [ 3125.193715] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3125.215062] LustreError: 103698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3125.730569] Lustre: server umount lustre-MDT0001 complete [ 3138.915138] Lustre: server umount lustre-OST0000 complete [ 3151.942073] Lustre: server umount lustre-OST0001 complete [ 3157.513419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3168.368967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3177.208530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3187.689584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3208.948752] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 3220.149690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3220.172696] Lustre: lustre-MDT0000: reset Object Index mappings [ 3220.742410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3226.140553] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3236.341536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3236.367722] Lustre: lustre-MDT0001: reset Object Index mappings [ 3242.066625] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3245.128216] Lustre: 115882:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3252.966564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3260.550889] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3262.055621] LustreError: 116235:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3262.103729] LustreError: 116235:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3262.151965] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:195 to 0x280000400:225) [ 3265.141983] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:418) [ 3270.780150] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3276.815763] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:161 to 0x2c0000401:193) [ 3276.818133] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3278.544522] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3287.750956] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3291.309843] Lustre: 117758:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3346.136742] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 07:33:06 (1787398386) [ 3374.062166] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3374.067645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3374.091788] Lustre: Skipped 3 previous similar messages [ 3374.108051] Lustre: Skipped 5 previous similar messages [ 3387.359305] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3387.743214] Lustre: server umount lustre-MDT0000 complete [ 3391.837277] LustreError: 114714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787398434 with bad export cookie 4981233687544510845 [ 3391.846741] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3391.853547] LustreError: 114714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3394.529981] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3394.540367] Lustre: Skipped 1 previous similar message [ 3406.304460] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3406.650246] Lustre: server umount lustre-MDT0001 complete [ 3411.218560] LustreError: 118521:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0000_UUID: namespace resource [0x280000401:0xfe:0x0].0x0 (ffff92c0c20d6c00) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 3417.615635] Lustre: server umount lustre-OST0000 complete [ 3427.778678] Lustre: server umount lustre-OST0001 complete [ 3435.229670] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 3446.948489] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 3487.658651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3488.188427] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3488.284232] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3488.522239] Lustre: lustre-MDT0000: new disk, initializing [ 3488.660203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3488.669643] Lustre: Skipped 3 previous similar messages [ 3488.708675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3493.507130] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3501.575348] Lustre: server umount lustre-MDT0000 complete [ 3512.266025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3512.549834] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3512.561772] Lustre: 122033:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3512.569722] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3517.725254] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3522.660749] Lustre: server umount lustre-MDT0000 complete [ 3531.037473] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 07:36:12 (1787398572) [ 3532.710722] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3534.544639] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 07:36:15 (1787398575) [ 3555.242919] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 3568.712788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3576.067152] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3586.528197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3586.670382] Lustre: 124083:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3586.768720] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3586.776936] Lustre: Skipped 1 previous similar message [ 3586.934523] Lustre: lustre-MDT0001: new disk, initializing [ 3587.199092] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3587.220661] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3593.149561] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3598.921604] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3606.093590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3606.441440] Lustre: lustre-OST0000: new disk, initializing [ 3606.448288] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3606.456360] Lustre: 125663:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3607.906052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3607.917900] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3608.080706] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3614.606536] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3627.781343] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3628.134638] Lustre: lustre-OST0001: new disk, initializing [ 3628.143501] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3628.158565] Lustre: 126535:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3629.653585] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3629.676451] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3629.907534] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3637.543604] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3649.749485] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3653.926298] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4096.031844] Lustre: 132978:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4096.037519] Lustre: 132978:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4096.042712] Lustre: 132978:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4096.047942] Lustre: 132978:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4096.054103] Lustre: 132978:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4096.069978] Lustre: 132978:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4404.461885] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 07:50:44 (1787399444) [ 4421.605908] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4421.611139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4421.631209] Lustre: Skipped 2 previous similar messages [ 4421.641917] Lustre: Skipped 17 previous similar messages [ 4426.726149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4426.739131] Lustre: Skipped 3 previous similar messages [ 4433.376129] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4433.619941] Lustre: server umount lustre-MDT0000 complete [ 4436.963380] LustreError: 127348:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4436.985058] LustreError: 127348:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 17 previous similar messages [ 4437.570812] LustreError: 124077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787399479 with bad export cookie 4981233687544579886 [ 4437.571674] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4437.581960] LustreError: 124077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4442.084307] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4442.105534] Lustre: Skipped 1 previous similar message [ 4442.109043] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4442.127470] Lustre: Skipped 5 previous similar messages [ 4444.678523] Lustre: server umount lustre-MDT0001 complete [ 4455.901488] Lustre: server umount lustre-OST0000 complete [ 4466.020734] Lustre: server umount lustre-OST0001 complete [ 4473.439083] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 4484.162607] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 4519.319630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4519.684761] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4519.732892] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4519.847616] Lustre: lustre-MDT0000: new disk, initializing [ 4520.033697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4520.036777] Lustre: Skipped 5 previous similar messages [ 4520.078311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4524.917731] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4537.931563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4538.128684] Lustre: 136037:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4538.208517] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4538.211409] Lustre: Skipped 1 previous similar message [ 4538.403884] Lustre: lustre-MDT0001: new disk, initializing [ 4538.726636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4538.742655] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4543.734302] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4555.697879] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4561.914689] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4569.417493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4569.681468] Lustre: lustre-OST0000: new disk, initializing [ 4569.684441] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4569.690692] Lustre: 138009:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4571.114550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4571.120175] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4571.229884] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4576.146156] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4585.349380] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4594.740163] Lustre: lustre-MDD0000: changelog on [ 4598.408813] Lustre: lustre-MDD0001: changelog on [ 6334.891562] Lustre: lustre-MDD0001: changelog off [ 6336.780418] Lustre: 136917:0:(llog_cat.c:984:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6337.675798] Lustre: lustre-MDD0000: changelog off [ 6347.749028] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6347.758383] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6347.767183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6347.774122] Lustre: Skipped 1 previous similar message [ 6347.797044] Lustre: Skipped 1 previous similar message [ 6350.251411] Lustre: server umount lustre-OST0000 complete [ 6356.448766] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6356.454149] LustreError: Skipped 1 previous similar message [ 6356.458337] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6356.474096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6360.847599] Lustre: server umount lustre-MDT0000 complete [ 6363.114345] LustreError: 136918:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6363.144538] LustreError: 136918:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6365.068961] LustreError: 136029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787401407 with bad export cookie 4981233687545403429 [ 6365.073629] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6365.078551] LustreError: 136029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6365.447800] Lustre: server umount lustre-MDT0001 complete [ 6378.278851] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 6381.237723] Key type lgssc unregistered [ 6381.588649] LNet: 147132:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6381.594701] LNetError: 147132:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6381.611957] LNet: Removed LNI 192.168.201.156@tcp [ 6382.540087] Key type .llcrypt unregistered [ 6382.547759] Key type ._llcrypt unregistered [ 6404.965644] Key type ._llcrypt registered [ 6404.967852] Key type .llcrypt registered [ 6405.136239] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 6423.847682] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 6425.075426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6425.225550] alg: No test for adler32 (adler32-zlib) [ 6426.500948] Lustre: Lustre: Build Version: 2.17.54_108_g575fb00 [ 6426.891870] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 6428.609354] Key type lgssc registered [ 6429.866419] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6469.711190] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6469.748741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6471.063180] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6471.100921] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6471.159995] Lustre: lustre-MDT0000: new disk, initializing [ 6471.255180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6471.294529] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6475.889663] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 6488.290776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6488.421304] Lustre: 150335:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6488.483771] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6488.487949] Lustre: Skipped 1 previous similar message [ 6488.566191] Lustre: lustre-MDT0001: new disk, initializing [ 6488.693660] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6488.750675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6488.781250] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6493.568476] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 6503.943354] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6510.673069] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6517.422368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6517.722514] Lustre: lustre-OST0000: new disk, initializing [ 6517.728844] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6517.737507] Lustre: 152308:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6517.834705] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6524.035099] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 6526.523365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6526.530436] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6526.622716] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6533.278166] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6539.623854] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_module obdecho/obdecho [ 6541.617121] LustreError: 153311:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6541.628349] LustreError: 153311:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6543.422748] LustreError: 153407:0:(echo_client.c:1883:echo_md_handler()) Can not resolve the path : rc = -2 [ 6555.104845] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6555.116051] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6555.136289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6557.664582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6557.668143] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6557.675780] Lustre: Skipped 1 previous similar message [ 6557.686984] Lustre: Skipped 1 previous similar message [ 6560.847860] Lustre: server umount lustre-MDT0000 complete [ 6564.588847] LustreError: 150328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787401606 with bad export cookie 12663632465884092518 [ 6564.589609] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6564.595703] LustreError: 150328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6564.880926] Lustre: server umount lustre-MDT0001 complete [ 6578.149145] Lustre: server umount lustre-OST0000 complete [ 6604.804646] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 6616.874316] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 6653.614136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6653.914532] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6653.957871] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6654.069258] Lustre: lustre-MDT0000: new disk, initializing [ 6654.158640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6654.172760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6659.626520] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 6673.748977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6673.865597] Lustre: 157122:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6673.906775] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6673.910763] Lustre: Skipped 1 previous similar message [ 6673.972890] Lustre: lustre-MDT0001: new disk, initializing [ 6674.053701] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6674.099302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6674.117956] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6679.528413] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 6690.507723] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6696.343374] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6703.191047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6703.494992] Lustre: lustre-OST0000: new disk, initializing [ 6703.500463] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6703.509942] Lustre: 159093:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6703.654346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6709.809183] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6709.818747] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6709.983363] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6710.278440] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 6719.880484] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6724.654537] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6724.945168] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6729.983709] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6730.246322] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6735.334466] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6735.352646] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6735.393522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6735.402378] Lustre: Skipped 1 previous similar message [ 6738.787560] Lustre: server umount lustre-OST0000 complete [ 6746.080462] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6746.110333] Lustre: Skipped 1 previous similar message [ 6746.124883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6746.137912] Lustre: Skipped 2 previous similar messages [ 6751.201889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6751.212058] Lustre: Skipped 1 previous similar message [ 6757.344676] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6757.629416] Lustre: server umount lustre-MDT0000 complete [ 6761.440742] LustreError: 157135:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6761.455482] LustreError: 157135:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6761.586431] LustreError: 157117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787401803 with bad export cookie 12663632465884093477 [ 6761.589419] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6761.613276] LustreError: 157117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6762.193986] Lustre: server umount lustre-MDT0001 complete [ 6766.142300] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 08:30:06 (1787401806) [ 6768.061685] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6769.897733] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 08:30:10 (1787401810) [ 6778.462983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6779.265889] LustreError: 161506:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6779.387560] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6784.487511] LustreError: 161507:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6785.228756] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 6789.605662] LustreError: 161506:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6794.722797] LustreError: 161507:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6797.309188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6803.214283] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 6810.974908] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6817.322996] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6825.240063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6825.891715] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6825.897634] Lustre: Skipped 1 previous similar message [ 6832.892483] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 6842.745493] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6863.899825] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.56@tcp (stopping) [ 6863.903701] Lustre: Skipped 2 previous similar messages [ 6864.362426] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6864.374062] LustreError: Skipped 1 previous similar message [ 6864.387050] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6864.419028] Lustre: Skipped 1 previous similar message [ 6865.182796] Lustre: server umount lustre-MDT0000 complete [ 6868.973829] LustreError: 161501:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6869.251682] LustreError: 162242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787401911 with bad export cookie 12663632465884094359 [ 6869.266217] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6869.277495] LustreError: 162242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6875.657981] Lustre: server umount lustre-MDT0001 complete [ 6883.582031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6899.529257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6899.863215] LustreError: 165368:0:(llog.c:1655:llog_backup()) MGC192.168.201.156@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6899.874119] Lustre: 165368:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.156@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6903.781707] LustreError: 165374:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6903.816274] LustreError: 165374:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6939.617542] LustreError: 165374:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6939.626583] LustreError: 165374:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 26 previous similar messages [ 6960.607386] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6960.617524] Lustre: 165368:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.156@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7001.598482] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7005.177776] LustreError: 165375:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7005.193746] LustreError: 165375:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 42 previous similar messages [ 7006.709171] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7007.157706] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7017.213137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7029.222178] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7039.468156] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xafbe42a2072038f0 [ 7039.484413] Lustre: MGC192.168.201.156@tcp: Connection restored to 0@lo (at 0@lo) [ 7045.102392] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7045.233883] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7054.080723] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7055.657102] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7061.908114] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7064.017756] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7080.929940] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7080.930027] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7080.943128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7080.943136] Lustre: Skipped 6 previous similar messages [ 7080.977034] Lustre: Skipped 4 previous similar messages [ 7086.334244] Lustre: server umount lustre-MDT0000 complete [ 7090.075878] LustreError: 162242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787402132 with bad export cookie 12663632465884100848 [ 7090.082633] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7090.094562] LustreError: 162242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7090.622951] Lustre: server umount lustre-MDT0001 complete [ 7105.048161] Lustre: server umount lustre-OST0000 complete [ 7126.748350] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 7141.081693] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 7179.622889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7179.909388] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7179.955080] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7180.034262] Lustre: lustre-MDT0000: new disk, initializing [ 7180.139263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7180.151307] Lustre: Skipped 1 previous similar message [ 7180.181972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7186.196547] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7199.636918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7199.787360] Lustre: 170478:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7200.062510] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7205.542298] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7215.313781] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7220.866283] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7227.994122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7228.248526] Lustre: lustre-OST0000: new disk, initializing [ 7228.251549] Lustre: Skipped 1 previous similar message [ 7228.258519] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7228.264965] Lustre: Skipped 2 previous similar messages [ 7228.274606] Lustre: 172446:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7235.029993] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7236.187243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7236.201615] Lustre: Skipped 1 previous similar message [ 7236.214784] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7236.376191] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7244.670946] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7249.980655] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7250.201866] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7254.921734] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7255.187716] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7262.183387] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7262.205459] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7262.230821] Lustre: Skipped 1 previous similar message [ 7262.240269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7262.246959] Lustre: Skipped 5 previous similar messages [ 7263.529909] Lustre: server umount lustre-OST0000 complete [ 7273.733573] Lustre: server umount lustre-MDT0000 complete [ 7277.539294] LustreError: 170486:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7277.558210] LustreError: 170486:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 31 previous similar messages [ 7277.644787] LustreError: 170472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787402319 with bad export cookie 12663632465884117655 [ 7277.652465] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7277.656693] LustreError: 170472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7278.038183] Lustre: server umount lustre-MDT0001 complete [ 7287.111836] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 08:38:47 (1787402327) [ 7294.196756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7299.878454] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7310.792855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7317.065869] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7326.646438] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7332.843933] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7340.601745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7348.483091] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7355.330967] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7376.753550] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 08:40:17 (1787402417) [ 7382.496607] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7382.513428] LustreError: Skipped 1 previous similar message [ 7382.515470] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7382.544817] Lustre: Skipped 3 previous similar messages [ 7382.554706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7382.564716] Lustre: Skipped 5 previous similar messages [ 7386.864840] Lustre: server umount lustre-OST0000 complete [ 7396.742830] Lustre: server umount lustre-MDT0000 complete [ 7400.822937] LustreError: 174837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787402443 with bad export cookie 12663632465884118530 [ 7400.853733] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7401.366222] Lustre: server umount lustre-MDT0001 complete [ 7414.110662] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 7417.232822] Key type lgssc unregistered [ 7417.590903] LNet: 178612:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7417.605875] LNetError: 178612:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7418.666323] LNet: Removed LNI 192.168.201.156@tcp [ 7419.646916] Key type .llcrypt unregistered [ 7419.650718] Key type ._llcrypt unregistered [ 7443.616616] Key type ._llcrypt registered [ 7443.620148] Key type .llcrypt registered [ 7443.713414] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 7445.426888] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7445.464419] alg: No test for adler32 (adler32-zlib) [ 7446.707716] Lustre: Lustre: Build Version: 2.17.54_108_g575fb00 [ 7447.031684] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 7448.759304] Key type lgssc registered [ 7450.359855] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7458.150502] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7458.168889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7473.952441] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7479.071988] LustreError: 179856:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.156@tcp: failed processing log, type 4: rc = -110 [ 7484.191641] LustreError: 179856:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7484.207399] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7484.213797] LustreError: MGC192.168.201.156@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7484.232720] LustreError: 179856:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7484.255596] LustreError: 179856:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7484.266398] LustreError: 179856:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7484.355950] Lustre: server umount lustre-OST0000 complete [ 7484.360735] LustreError: 179856:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7494.359841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7495.832827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7500.696717] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7506.399935] LustreError: 180357:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7506.421370] LustreError: 180357:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7509.636607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7509.999101] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7515.088920] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7522.666344] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7529.472776] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7537.234839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7537.453448] LustreError: 182172:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7537.461319] Lustre: 182172:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.201.156@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7537.613717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7544.694765] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7551.961542] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7555.171866] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7555.187845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7555.198628] Lustre: Skipped 1 previous similar message [ 7555.229130] Lustre: Skipped 2 previous similar messages [ 7560.593463] Lustre: server umount lustre-MDT0000 complete [ 7560.674097] LustreError: 180356:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7560.703555] LustreError: 180356:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7564.855552] LustreError: 180336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787402607 with bad export cookie 17588702141297552016 [ 7564.871045] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7564.879089] LustreError: 180336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7565.495397] LustreError: 181085:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000def86410 x1874227352334208/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7565.841958] Lustre: server umount lustre-MDT0001 complete [ 7581.517225] Lustre: server umount lustre-OST0000 complete [ 7593.789245] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 7605.643515] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 7648.428985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7648.706397] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7648.741814] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7648.824340] Lustre: lustre-MDT0000: new disk, initializing [ 7648.887120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7648.905189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7653.484501] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7664.778993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7664.911499] Lustre: 185698:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7664.967376] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7664.971222] Lustre: Skipped 1 previous similar message [ 7665.085275] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7665.111733] Lustre: lustre-MDT0001: new disk, initializing [ 7665.209306] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7665.259360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7665.271165] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7670.383240] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7680.613444] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7688.692356] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7698.868202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7699.269562] Lustre: lustre-OST0000: new disk, initializing [ 7699.278604] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7699.290399] Lustre: 187666:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7699.452964] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7706.729525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7706.739907] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7706.826034] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7706.901776] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7717.556995] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7724.845436] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7725.148588] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7731.495454] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7731.773400] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7737.825316] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7737.827614] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7737.850731] Lustre: Skipped 1 previous similar message [ 7737.868704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7737.885336] Lustre: Skipped 1 previous similar message [ 7740.272758] Lustre: server umount lustre-OST0000 complete [ 7747.042218] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7747.052456] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7747.069579] Lustre: Skipped 1 previous similar message [ 7747.073418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7750.575833] Lustre: server umount lustre-MDT0000 complete [ 7753.190607] LustreError: 187908:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7753.199923] LustreError: 187908:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7754.626658] LustreError: 185688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787402796 with bad export cookie 17588702141297552737 [ 7754.643186] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7754.645284] LustreError: 185688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7754.916576] Lustre: server umount lustre-MDT0001 complete [ 7766.983366] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 08:46:47 (1787402807) [ 7789.800225] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 7792.351139] Key type lgssc unregistered [ 7792.647439] LNet: 190616:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7792.663553] LNetError: 190616:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7792.680120] LNet: Removed LNI 192.168.201.156@tcp [ 7793.510392] Key type .llcrypt unregistered [ 7793.512472] Key type ._llcrypt unregistered [ 7813.940842] Key type ._llcrypt registered [ 7813.947236] Key type .llcrypt registered [ 7814.096878] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 7815.436930] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7815.596500] alg: No test for adler32 (adler32-zlib) [ 7816.869221] Lustre: Lustre: Build Version: 2.17.54_108_g575fb00 [ 7817.221902] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 7819.055714] Key type lgssc registered [ 7820.214457] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7827.917970] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7827.964113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7829.538969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7834.660800] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7838.706088] LustreError: 191837:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7838.719976] LustreError: 191837:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7843.812680] LustreError: 191838:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7845.159141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7845.444318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7851.196432] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7859.130444] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7865.164234] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7874.277977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7874.574970] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7874.579058] LustreError: 193701:0:(llog.c:1696:llog_backup()) MGC192.168.201.156@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7874.590736] Lustre: 193701:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.201.156@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7874.766396] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7881.809957] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7891.258308] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7895.521735] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7895.538179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7895.543622] Lustre: Skipped 1 previous similar message [ 7895.557987] Lustre: Skipped 2 previous similar messages [ 7899.783186] Lustre: server umount lustre-MDT0000 complete [ 7900.653795] LustreError: 191832:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7900.666033] LustreError: 191832:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7903.881345] LustreError: 191817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787402946 with bad export cookie 8864575255077941275 [ 7903.898572] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7904.321895] Lustre: server umount lustre-MDT0001 complete [ 7917.983259] Lustre: server umount lustre-OST0000 complete [ 7927.147345] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 7937.153563] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 7978.051940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7978.326217] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7978.377339] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7978.468196] Lustre: lustre-MDT0000: new disk, initializing [ 7978.552965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7978.566657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7984.301358] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7997.431737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7997.637044] Lustre: 197224:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7997.721716] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7997.724190] Lustre: Skipped 1 previous similar message [ 7997.729861] LustreError: 198007:0:(llog.c:1696:llog_backup()) MGC192.168.201.156@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7997.741383] LustreError: 198007:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7997.748818] Lustre: 198007:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.156@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7997.807943] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7997.812616] Lustre: Skipped 1 previous similar message [ 7997.950111] Lustre: lustre-MDT0001: new disk, initializing [ 7998.174481] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7998.211291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7998.221966] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8003.151541] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8013.315581] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8020.703862] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8027.691689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8027.929121] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8027.930963] Lustre: Skipped 1 previous similar message [ 8027.933607] LustreError: 199170:0:(llog.c:1696:llog_backup()) MGC192.168.201.156@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8027.943039] LustreError: 199170:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8027.947723] Lustre: 199170:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.156@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8027.954151] Lustre: 199170:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8027.987901] Lustre: lustre-OST0000: new disk, initializing [ 8027.994264] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8028.001774] Lustre: 199196:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8028.127525] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8034.415332] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8034.439258] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8034.568913] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8035.250851] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8046.312591] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8052.052431] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8052.348435] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8057.952763] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8058.266435] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8065.506727] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8065.512828] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8065.521776] LustreError: Skipped 1 previous similar message [ 8065.533541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8065.547550] Lustre: Skipped 2 previous similar messages [ 8065.557083] Lustre: Skipped 1 previous similar message [ 8066.792393] Lustre: server umount lustre-OST0000 complete [ 8075.232911] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8075.251711] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8075.276238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8077.125684] Lustre: server umount lustre-MDT0000 complete [ 8080.867895] LustreError: 197236:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8080.897259] LustreError: 197236:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8082.019212] LustreError: 197218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787403124 with bad export cookie 8864575255077941989 [ 8082.031153] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8082.042664] LustreError: 197218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 8082.630902] Lustre: server umount lustre-MDT0001 complete [ 8094.352386] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 08:52:14 (1787403134) [ 8113.627143] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 8124.332754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8124.614685] LustreError: 202325:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8124.642668] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8129.980563] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8134.138553] LustreError: 202326:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8134.167591] LustreError: 202326:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8139.247550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8144.100889] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8147.038439] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8155.436836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8163.029291] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8172.182850] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8172.334476] Lustre: lustre-OST0001: new disk, initializing [ 8172.338307] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8172.345379] Lustre: 204637:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8172.425290] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8172.434212] Lustre: Skipped 2 previous similar messages [ 8178.254099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8178.265986] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8178.388966] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8178.497532] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8188.701910] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8195.900267] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8198.670513] Lustre: DEBUG MARKER: ADD OST3 [ 8205.517786] loop: module loaded [ 8205.626655] loop0: detected capacity change from 0 to 204800000 [ 8205.897431] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8205.918801] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8205.946641] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8213.536580] loop0: detected capacity change from 0 to 204800000 [ 8213.572749] Lustre: DEBUG MARKER: STOP OST3 [ 8221.161182] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8221.326263] Lustre: lustre-OST0003: new disk, initializing [ 8221.329709] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8221.336696] Lustre: 206706:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8223.135667] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8223.270794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8223.277432] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8223.343088] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8224.174491] Lustre: Failing over lustre-OST0003 [ 8224.399107] Lustre: server umount lustre-OST0003 complete [ 8225.188065] LustreError: 206850:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8228.328782] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8228.335710] Lustre: Skipped 1 previous similar message [ 8228.336975] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8228.348167] LustreError: 203814:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8228.368704] LustreError: 203814:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8228.571591] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8239.743619] Lustre: DEBUG MARKER: START OST3 again [ 8246.327272] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8246.784053] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8246.786897] Lustre: Skipped 1 previous similar message [ 8246.812907] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8248.224938] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8248.826801] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8248.831235] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8253.385664] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8275.855152] Lustre: Failing over lustre-OST0003 [ 8275.962754] Lustre: server umount lustre-OST0003 complete [ 8277.503594] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8277.521305] LustreError: 203812:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8277.530703] Lustre: Skipped 2 previous similar messages [ 8277.568234] LustreError: 203812:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 8297.961517] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8297.982921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8297.985271] Lustre: Skipped 1 previous similar message [ 8298.006349] Lustre: Skipped 2 previous similar messages [ 8301.278680] Lustre: server umount lustre-MDT0000 complete [ 8306.392532] LustreError: 202306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787403348 with bad export cookie 8864575255077942864 [ 8306.403292] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8306.411496] LustreError: 202306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8308.216739] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8313.191673] Lustre: server umount lustre-MDT0001 complete [ 8317.847824] Lustre: server umount lustre-OST0000 complete [ 8322.445787] Lustre: server umount lustre-OST0001 complete [ 8328.942475] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 8338.257644] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 8375.312887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8375.525465] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8375.554989] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8375.658725] Lustre: lustre-MDT0000: new disk, initializing [ 8375.736214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8375.756040] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8381.334860] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8394.138347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8394.307726] Lustre: 212250:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8394.365916] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8394.369995] Lustre: Skipped 1 previous similar message [ 8394.503144] Lustre: lustre-MDT0001: new disk, initializing [ 8394.746121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8394.763606] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8399.818598] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8411.411631] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8417.521457] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8424.194037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8424.461346] Lustre: 214219:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8426.078568] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8426.173637] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8430.276838] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8440.163305] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8445.379030] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8445.629426] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8451.427648] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8451.649579] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8457.186802] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8457.189774] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8457.208893] LustreError: Skipped 1 previous similar message [ 8457.209897] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8457.209903] Lustre: Skipped 3 previous similar messages [ 8457.293639] Lustre: Skipped 5 previous similar messages [ 8460.281579] Lustre: server umount lustre-OST0000 complete [ 8466.400380] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8470.934455] Lustre: server umount lustre-MDT0000 complete [ 8472.546140] LustreError: 212262:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8472.570116] LustreError: 212262:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 17 previous similar messages [ 8474.745756] LustreError: 212242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787403517 with bad export cookie 8864575255077945160 [ 8474.754079] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8474.759057] LustreError: 212242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8475.306367] Lustre: server umount lustre-MDT0001 complete [ 8479.638399] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 08:58:40 (1787403520) [ 8496.078930] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 8506.027354] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 8542.666386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8542.962505] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8543.022730] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8543.032876] Lustre: Skipped 1 previous similar message [ 8543.135179] Lustre: lustre-MDT0000: new disk, initializing [ 8543.138381] Lustre: Skipped 1 previous similar message [ 8543.247924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8543.256751] Lustre: Skipped 1 previous similar message [ 8548.889449] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8559.246271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8559.365325] Lustre: 218830:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8559.631603] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8564.408114] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8574.429860] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8580.092446] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8587.444351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8587.855822] Lustre: 220800:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8589.949174] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8594.787962] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8604.148541] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8608.900615] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8609.152738] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8614.278936] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8614.565517] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8620.514160] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8620.517377] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8620.524110] LustreError: Skipped 1 previous similar message [ 8620.532035] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8620.542201] Lustre: Skipped 3 previous similar messages [ 8620.560275] Lustre: Skipped 5 previous similar messages [ 8623.075860] Lustre: server umount lustre-OST0000 complete [ 8633.195139] Lustre: server umount lustre-MDT0000 complete [ 8635.876775] LustreError: 218844:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8635.899317] LustreError: 218844:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8637.622938] LustreError: 219626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787403679 with bad export cookie 8864575255077946049 [ 8637.629626] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8637.637547] LustreError: 219626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8637.873272] Lustre: server umount lustre-MDT0001 complete [ 8646.603482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8646.979207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8646.990712] Lustre: Skipped 5 previous similar messages [ 8651.139243] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8659.158727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8663.372934] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8671.616375] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8677.573526] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8684.318906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8690.282938] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8697.165175] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8707.044219] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8707.047165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8707.048561] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8707.054168] Lustre: Skipped 3 previous similar messages [ 8707.064915] Lustre: Skipped 3 previous similar messages [ 8712.671390] Lustre: server umount lustre-OST0000 complete [ 8722.288133] Lustre: server umount lustre-MDT0000 complete [ 8725.634465] LustreError: 223007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787403767 with bad export cookie 8864575255077946931 [ 8725.642546] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8725.923582] Lustre: server umount lustre-MDT0001 complete [ 8838.476514] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 8841.087570] Key type lgssc unregistered [ 8841.423940] LNet: 226691:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8841.442688] LNetError: 226691:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8842.476507] LNet: Removed LNI 192.168.201.156@tcp [ 8843.714169] Key type .llcrypt unregistered [ 8843.715547] Key type ._llcrypt unregistered [ 8869.475315] Key type ._llcrypt registered [ 8869.477974] Key type .llcrypt registered [ 8869.701438] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 8885.818144] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 8887.451584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8887.476229] alg: No test for adler32 (adler32-zlib) [ 8888.477186] Lustre: Lustre: Build Version: 2.17.54_108_g575fb00 [ 8888.765615] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 8890.487319] Key type lgssc registered [ 8891.864853] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8928.373321] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8928.403326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8929.781566] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8929.820653] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8929.884972] Lustre: lustre-MDT0000: new disk, initializing [ 8929.978825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8930.011614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8934.821530] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8945.673770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8945.855130] Lustre: 230045:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8945.936731] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8945.943520] Lustre: Skipped 1 previous similar message [ 8946.036152] Lustre: lustre-MDT0001: new disk, initializing [ 8946.167444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8946.210764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8946.235596] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8951.140339] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8960.872360] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8967.017357] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8974.214471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8974.554751] Lustre: lustre-OST0000: new disk, initializing [ 8974.558611] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8974.564231] Lustre: 232013:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8974.658618] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8981.433954] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 8982.109635] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8982.119131] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8982.239091] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8991.408933] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8996.110448] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8996.312349] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9001.186345] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9001.405257] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9008.098680] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9008.099690] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9008.118554] LustreError: Skipped 1 previous similar message [ 9008.121526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9008.169147] Lustre: Skipped 1 previous similar message [ 9013.248203] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9013.259428] Lustre: Skipped 2 previous similar messages [ 9018.335485] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9018.344754] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9018.364286] Lustre: Skipped 1 previous similar message [ 9018.686500] Lustre: server umount lustre-OST0000 complete [ 9023.456471] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9023.457287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9023.463842] Lustre: Skipped 1 previous similar message [ 9023.474153] Lustre: Skipped 1 previous similar message [ 9028.583073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9029.471302] Lustre: server umount lustre-MDT0000 complete [ 9033.377812] LustreError: 230036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787404075 with bad export cookie 16335431222366252447 [ 9033.390178] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9033.390792] LustreError: 230036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9033.752512] Lustre: server umount lustre-MDT0001 complete [ 9052.912465] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 9064.200192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9064.531050] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9069.351507] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9069.542416] LustreError: 234947:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9069.569870] LustreError: 234947:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9074.658379] LustreError: 234948:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9077.480130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9077.779665] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9082.171519] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9085.470758] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9091.949897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9092.451371] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9098.754046] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9107.204326] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9107.368931] Lustre: lustre-OST0001: new disk, initializing [ 9107.374159] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9107.383719] Lustre: 237254:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9113.227906] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9115.302430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9115.320681] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9115.422960] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9124.959858] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9132.465829] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9147.130858] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 09:09:47 (1787404187) [ 9149.417117] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9149.424233] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9149.459069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9149.468205] Lustre: Skipped 1 previous similar message [ 9151.464453] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9151.474125] Lustre: Skipped 2 previous similar messages [ 9155.631479] Lustre: server umount lustre-MDT0000 complete [ 9156.604687] LustreError: 234943:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9156.617793] LustreError: 234943:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9160.135453] LustreError: 234930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787404202 with bad export cookie 16335431222366253329 [ 9160.147795] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9160.153038] LustreError: 234930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 9160.654858] Lustre: server umount lustre-MDT0001 complete [ 9176.545273] Lustre: server umount lustre-OST0000 complete [ 9178.277257] Lustre: 228269:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787404204/real 1787404204] req@ffff92bf8e3e3b80 x1874228864466432/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787404220 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9178.296510] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9179.998858] Lustre: server umount lustre-OST0001 complete [ 9186.281236] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 9196.960556] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 9231.867201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9232.216086] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9232.247219] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9232.322667] Lustre: lustre-MDT0000: new disk, initializing [ 9232.434442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9232.439532] Lustre: Skipped 1 previous similar message [ 9232.452141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9236.590666] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9246.777942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9246.916066] Lustre: 241197:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9247.010884] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9247.016260] Lustre: Skipped 1 previous similar message [ 9247.120394] Lustre: lustre-MDT0001: new disk, initializing [ 9247.276631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9247.299207] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9252.486140] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9262.765516] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9269.038854] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9276.152418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9276.538698] Lustre: lustre-OST0000: new disk, initializing [ 9276.542271] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9276.551917] Lustre: 243169:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9276.650269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9276.655077] Lustre: Skipped 1 previous similar message [ 9278.526903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9278.532593] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9278.681709] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9283.506534] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9292.845959] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9298.032617] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9298.209948] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9303.103336] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9303.349415] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9309.666646] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9309.670832] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9309.672227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9309.672234] Lustre: Skipped 4 previous similar messages [ 9309.677233] LustreError: Skipped 1 previous similar message [ 9309.726979] Lustre: Skipped 1 previous similar message [ 9311.730614] Lustre: server umount lustre-OST0000 complete [ 9318.882553] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9322.047636] Lustre: server umount lustre-MDT0000 complete [ 9325.025098] LustreError: 241205:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9325.050549] LustreError: 241205:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9325.514061] LustreError: 241191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787404367 with bad export cookie 16335431222366255464 [ 9325.522607] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9325.523947] LustreError: 241191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9325.798381] Lustre: server umount lustre-MDT0001 complete [ 9333.597155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9333.957087] LustreError: 245386:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9338.851163] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9344.486604] LustreError: 245387:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9344.501383] LustreError: 245387:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9347.582383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9347.948488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9347.952041] Lustre: Skipped 1 previous similar message [ 9352.631855] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9359.343222] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9364.933785] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9373.106538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9380.450488] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9387.956758] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9538.533635] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9538.540591] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9538.544362] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9538.544370] Lustre: Skipped 4 previous similar messages [ 9538.568207] Lustre: Skipped 3 previous similar messages [ 9540.043677] Lustre: server umount lustre-OST0000 complete [ 9547.748202] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9551.087836] Lustre: server umount lustre-MDT0000 complete [ 9553.889248] LustreError: 245382:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9553.914194] LustreError: 245382:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9555.904354] LustreError: 245367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787404598 with bad export cookie 16335431222366256346 [ 9555.907796] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9555.917832] LustreError: 245367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9556.500767] Lustre: server umount lustre-MDT0001 complete [ 9656.652882] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 9659.626017] Key type lgssc unregistered [ 9659.932135] LNet: 248996:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9659.944102] LNetError: 248996:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9659.963152] LNet: Removed LNI 192.168.201.156@tcp [ 9660.725303] Key type .llcrypt unregistered [ 9660.728916] Key type ._llcrypt unregistered [ 9685.548435] Key type ._llcrypt registered [ 9685.551760] Key type .llcrypt registered [ 9685.638895] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 9700.171839] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 9701.296474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9701.338761] alg: No test for adler32 (adler32-zlib) [ 9702.395810] Lustre: Lustre: Build Version: 2.17.54_108_g575fb00 [ 9702.696834] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 9704.391131] Key type lgssc registered [ 9705.298768] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9755.725333] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 9765.720685] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9765.770295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9767.062125] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9767.090254] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9767.154533] Lustre: lustre-MDT0000: new disk, initializing [ 9767.218787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9767.231693] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9772.965913] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9784.457392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9784.681691] Lustre: 253087:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9784.815954] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9784.820849] Lustre: Skipped 1 previous similar message [ 9784.937555] Lustre: lustre-MDT0001: new disk, initializing [ 9785.123068] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9785.165678] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9785.181989] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9790.459659] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9796.916541] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9804.780229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9805.289802] Lustre: lustre-OST0000: new disk, initializing [ 9805.298573] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9805.305496] Lustre: 254717:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9805.452639] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9807.216686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9807.245683] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9807.443451] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9813.299320] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9825.707206] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9825.984796] Lustre: lustre-OST0001: new disk, initializing [ 9825.987597] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9826.008425] Lustre: 255590:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9826.194281] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9832.693324] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9832.708114] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9832.899150] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9833.586752] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9846.057869] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9855.863709] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9868.770381] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9868.778915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9868.789054] Lustre: Skipped 2 previous similar messages [ 9868.805836] Lustre: Skipped 3 previous similar messages [ 9873.880462] Lustre: server umount lustre-MDT0000 complete [ 9877.528202] LustreError: 256357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787404919 with bad export cookie 18186274318563699660 [ 9877.531664] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9877.540569] LustreError: 256357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9877.887224] Lustre: server umount lustre-MDT0001 complete [ 9892.147069] Lustre: server umount lustre-OST0000 complete [ 9895.393905] Lustre: 250572:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787404921/real 1787404921] req@ffff92c08cfe0380 x1874229717975040/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787404937 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9895.419205] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9895.439949] Lustre: Skipped 1 previous similar message [ 9895.765316] Lustre: server umount lustre-OST0001 complete [ 9911.126869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9924.533939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9934.523158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9943.204975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9961.792518] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 9972.442123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9972.457607] Lustre: lustre-MDT0000: reset Object Index mappings [ 9972.720027] LustreError: 260602:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9972.741263] LustreError: 260602:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 9972.778811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9976.697378] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9977.834142] LustreError: 260603:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9982.955443] LustreError: 260602:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9984.822231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9984.842688] Lustre: lustre-MDT0001: reset Object Index mappings [ 9985.199420] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9989.002830] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 9991.763791] Lustre: 261745:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9998.375604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10005.040287] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10010.282179] LustreError: 262100:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10013.549624] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10013.815943] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10013.821457] Lustre: Skipped 1 previous similar message [10017.936439] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10017.940122] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10020.789304] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10027.412911] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10030.612800] Lustre: 263619:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10036.193647] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10036.205383] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10036.233540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10036.995631] Lustre: server umount lustre-MDT0000 complete [10038.773040] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10038.776052] LustreError: 261336:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10038.813498] Lustre: Skipped 1 previous similar message [10038.830756] LustreError: 261336:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10040.645068] LustreError: 260584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787405082 with bad export cookie 18186274318563706450 [10040.660092] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10040.978035] Lustre: server umount lustre-MDT0001 complete [10055.029681] Lustre: server umount lustre-OST0000 complete [10068.714947] Lustre: server umount lustre-OST0001 complete [10086.680298] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 09:25:27 (1787405127) [10100.350235] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [10110.213444] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [10142.169346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10142.384318] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10142.418728] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10142.492631] Lustre: lustre-MDT0000: new disk, initializing [10142.601473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10142.619482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10146.884497] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10157.588020] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10157.721782] Lustre: 267873:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10157.779892] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10157.784964] Lustre: Skipped 1 previous similar message [10157.858472] Lustre: lustre-MDT0001: new disk, initializing [10158.003848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10158.023556] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10162.477642] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10171.881260] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10176.808803] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10183.776929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10184.059259] Lustre: lustre-OST0000: new disk, initializing [10184.064652] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10184.068381] Lustre: 269838:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10185.522572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10185.536646] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10185.741994] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10191.241993] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10201.959727] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10207.967879] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10208.275687] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10214.047810] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10214.346717] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10216.432981] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10216.436104] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10216.454903] Lustre: Skipped 2 previous similar messages [10216.455573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10216.461858] LustreError: Skipped 1 previous similar message [10216.481140] Lustre: Skipped 1 previous similar message [10221.542966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10221.554254] Lustre: Skipped 1 previous similar message [10222.500764] Lustre: server umount lustre-OST0000 complete [10226.659655] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10226.670759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10226.684373] Lustre: Skipped 1 previous similar message [10226.702325] Lustre: Skipped 1 previous similar message [10232.325858] Lustre: server umount lustre-MDT0000 complete [10236.360943] LustreError: 267865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787405278 with bad export cookie 18186274318563708543 [10236.366666] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10236.372609] LustreError: 267865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [10237.102675] Lustre: server umount lustre-MDT0001 complete [10254.289526] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [10264.835434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10265.324918] LustreError: 272772:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10265.348746] LustreError: 272772:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10265.377030] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10265.380712] Lustre: Skipped 2 previous similar messages [10269.856710] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10277.691252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10282.484499] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10285.110764] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10291.185890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10297.401508] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10304.758434] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10305.016779] Lustre: lustre-OST0001: new disk, initializing [10305.024891] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10305.034367] Lustre: 275079:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10306.190968] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10306.205562] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10306.343209] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10311.543988] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10320.804691] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10324.479985] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10329.883385] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10330.075386] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10336.227515] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10336.229216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10336.253526] Lustre: Skipped 3 previous similar messages [10336.264395] Lustre: Skipped 5 previous similar messages [10341.674294] Lustre: server umount lustre-MDT0000 complete [10345.173977] LustreError: 272753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787405387 with bad export cookie 18186274318563709425 [10345.185338] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10345.186718] LustreError: 272753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10345.430301] Lustre: server umount lustre-MDT0001 complete [10359.395906] Lustre: server umount lustre-OST0000 complete [10362.336346] Lustre: 250572:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787405388/real 1787405388] req@ffff92bf8e3e3100 x1874229718124032/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787405404 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10363.701512] Lustre: server umount lustre-OST0001 complete [10387.086051] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [10400.577341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10401.271980] LustreError: 278136:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10401.307417] LustreError: 278136:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10401.423207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10401.426153] Lustre: Skipped 3 previous similar messages [10406.764486] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10415.939910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10420.466416] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10422.907870] Lustre: 279276:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10428.320763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10435.016790] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10440.887311] LustreError: 279632:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10440.918505] LustreError: 279632:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10443.307348] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10446.659322] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10446.698177] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10449.757394] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10456.880774] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10460.205827] Lustre: 281153:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10464.937480] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10464.988511] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10476.101991] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 09:31:56 (1787405516) [10492.902645] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10492.918339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10492.936649] Lustre: Skipped 4 previous similar messages [10492.945854] Lustre: Skipped 7 previous similar messages [10497.532926] Lustre: server umount lustre-MDT0000 complete [10500.998588] LustreError: 278117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787405543 with bad export cookie 18186274318563715732 [10501.003949] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10501.017504] LustreError: 278117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10505.187066] LustreError: 279652:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10505.207722] LustreError: 279652:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [10507.575467] Lustre: server umount lustre-MDT0001 complete [10517.307861] Lustre: server umount lustre-OST0000 complete [10521.562549] Lustre: server umount lustre-OST0001 complete [10527.738843] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [10537.376233] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [10581.971284] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [10595.719708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10595.932510] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10595.954905] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10596.022274] Lustre: lustre-MDT0000: new disk, initializing [10596.090727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10601.212738] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10611.371788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10611.493696] Lustre: 285391:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10611.692229] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10615.876439] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10626.008632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10626.277709] Lustre: 287025:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10627.802816] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10632.526399] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10643.598955] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10643.778886] Lustre: 287897:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10645.684711] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10645.702147] Lustre: Skipped 1 previous similar message [10645.813117] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10650.114622] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10659.279968] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10662.558936] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10662.563442] Lustre: Skipped 1 previous similar message [10681.989801] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 09:35:22 (1787405722) [10686.962359] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10686.969281] LustreError: 287578:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10686.980855] Lustre: Skipped 5 previous similar messages [10687.014629] LustreError: 287578:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10687.473268] Lustre: server umount lustre-MDT0000 complete [10692.149489] LustreError: 285384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787405734 with bad export cookie 18186274318563723607 [10692.154252] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10692.175240] LustreError: 285384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10692.652978] Lustre: server umount lustre-MDT0001 complete [10707.913877] Lustre: server umount lustre-OST0000 complete [10722.549375] Lustre: server umount lustre-OST0001 complete [10731.176612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10731.796258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10731.802534] Lustre: Skipped 7 previous similar messages [10736.583695] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10747.792277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10752.931337] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10761.884724] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10768.080718] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10775.485348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10782.388926] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10789.606601] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10808.955806] loop1: detected capacity change from 0 to 204800000 [10809.145553] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10809.166545] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10809.183521] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10813.761451] loop1: detected capacity change from 0 to 204800000 [10818.816508] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10818.934989] LustreError: 290131:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs22818: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10818.960041] LustreError: 290131:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs22818 [10818.971716] LustreError: 290131:0:(mgs_handler.c:517:mgs_target_reg()) fs22818-MDT0001: Error checking target index: rc = -13 [10818.984866] LustreError: fs22818-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10818.996657] LustreError: 293861:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10819.007147] LustreError: 293861:0:(tgt_mount.c:2129:server_put_super()) no obd fs22818-MDT0001 [10819.012385] LustreError: 293861:0:(tgt_mount.c:116:server_deregister_mount()) fs22818-MDT0001 not registered [10819.088239] Lustre: server umount fs22818-MDT0001 complete [10819.097462] LustreError: 293861:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [10827.932923] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10828.112314] Lustre: Setting parameter fs22818-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs22818-MDT0001 [10828.206784] Lustre: srv-fs22818-MDT0001: No data found on store. Initialize space. [10828.210387] Lustre: Skipped 4 previous similar messages [10828.353493] Lustre: fs22818-MDT0001: new disk, initializing [10828.360508] Lustre: Skipped 3 previous similar messages [10828.520855] Lustre: srv-fs22818-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10828.532470] Lustre: 294328:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs22818-MDT0001-mdtlov: can't create new llog: rc = -115 [10828.547144] LustreError: 294328:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs22818-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10833.121916] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10840.185203] Lustre: server umount fs22818-MDT0001 complete [10854.893207] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10855.300444] loop1: detected capacity change from 0 to 204800000 [10861.269267] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10861.809350] Lustre: 295896:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs22818-MDT0001-mdtlov: can't create new llog: rc = -115 [10861.824467] LustreError: 295896:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs22818-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10866.118757] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10870.816518] Lustre: server umount fs22818-MDT0001 complete [10875.376492] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10881.507910] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10881.522125] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10881.546197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10881.555624] Lustre: Skipped 4 previous similar messages [10886.852132] Lustre: server umount lustre-MDT0000 complete [10896.230404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10896.424863] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10901.718554] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10902.021237] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10909.438618] loop1: detected capacity change from 0 to 204800000 [10916.116119] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10916.622654] Lustre: 298343:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs22818-MDT0001-mdtlov: can't create new llog: rc = -115 [10916.638157] LustreError: 298343:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs22818-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10922.124765] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [10928.750978] Lustre: server umount fs22818-MDT0001 complete [10942.961551] LustreError: 291973:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10942.978279] LustreError: 291973:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 108 previous similar messages [10952.167908] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10952.186829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10952.189867] Lustre: Skipped 4 previous similar messages [10957.388659] Lustre: server umount lustre-OST0000 complete [10963.424826] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10963.443917] LustreError: Skipped 1 previous similar message [10975.200100] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10975.486495] Lustre: server umount lustre-MDT0000 complete [10980.184750] LustreError: 298837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787406022 with bad export cookie 18186274318563726288 [10980.190037] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10980.214335] LustreError: 298837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10980.729948] Lustre: server umount lustre-MDT0001 complete [10995.600740] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [10999.048723] Key type lgssc unregistered [10999.406595] LNet: 300691:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10999.415584] LNetError: 300691:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11000.497203] LNet: Removed LNI 192.168.201.156@tcp [11001.255161] Key type .llcrypt unregistered [11001.258653] Key type ._llcrypt unregistered [11024.055502] Key type ._llcrypt registered [11024.058284] Key type .llcrypt registered [11024.200292] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [11025.259619] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11025.356017] alg: No test for adler32 (adler32-zlib) [11026.728359] Lustre: Lustre: Build Version: 2.17.54_108_g575fb00 [11026.993493] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [11028.759177] Key type lgssc registered [11029.906850] Lustre: Echo OBD driver; http://www.lustre.org/ [11038.976114] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11038.997737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11040.493043] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11045.047268] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [11049.955042] LustreError: 301964:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11049.992806] LustreError: 301964:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11053.720727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11054.190252] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11059.053725] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [11067.752842] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11073.564625] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11081.189951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11081.637230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11084.728894] LustreError: 303793:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11087.600526] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [11089.898941] LustreError: 303975:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11095.026996] LustreError: 303793:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11095.648650] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11099.109250] LustreError: 303796:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11099.135153] LustreError: 303796:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11108.844153] LustreError: 303795:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11108.887816] LustreError: 303795:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11120.595273] loop1: detected capacity change from 0 to 204800000 [11120.923739] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11120.932802] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11120.942511] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11128.866262] loop1: detected capacity change from 0 to 204800000 [11129.333477] LustreError: 303795:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11129.369639] LustreError: 303795:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [11136.541759] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11136.922991] Lustre: fs7815-OST0000: new disk, initializing [11136.929498] Lustre: srv-fs7815-OST0000: No data found on store. Initialize space. [11136.944212] Lustre: 305703:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11137.040487] Lustre: fs7815-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11144.445084] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [11153.366663] Lustre: server umount fs7815-OST0000 complete [11165.189118] LustreError: 303793:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11165.220486] LustreError: 303793:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 20 previous similar messages [11165.683105] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11166.146775] loop1: detected capacity change from 0 to 204800000 [11172.009719] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11172.417602] Lustre: fs7815-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11178.722786] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [11181.038397] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11186.779234] loop2: detected capacity change from 0 to 204800000 [11186.937537] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11186.945671] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11186.953545] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11192.066792] loop2: detected capacity change from 0 to 204800000 [11197.672279] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11197.767938] LustreError: fs7815-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11197.789785] LustreError: fs7815-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11197.804205] LustreError: 308741:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11197.811450] LustreError: 308741:0:(tgt_mount.c:2129:server_put_super()) no obd fs7815-OST0001 [11197.819892] LustreError: 308741:0:(tgt_mount.c:116:server_deregister_mount()) fs7815-OST0001 not registered [11197.845941] Lustre: server umount fs7815-OST0001 complete [11197.849102] LustreError: 308741:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [11208.378224] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11208.516874] Lustre: fs7815-OST0001: new disk, initializing [11208.521414] Lustre: srv-fs7815-OST0001: No data found on store. Initialize space. [11208.531911] Lustre: 309294:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11208.639627] Lustre: fs7815-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11215.813883] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [11224.526108] Lustre: server umount fs7815-OST0000 complete [11231.726773] LustreError: 303793:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11231.752254] LustreError: 303793:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 38 previous similar messages [11235.263844] Lustre: server umount fs7815-OST0001 complete [11247.499431] loop1: detected capacity change from 0 to 204800000 [11253.228656] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11253.455726] Lustre: fs7815-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11261.180911] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [11269.050254] loop2: detected capacity change from 0 to 204800000 [11276.662959] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11285.046083] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [11296.524764] Lustre: server umount fs7815-OST0000 complete [11307.397346] Lustre: server umount fs7815-OST0001 complete [11319.266979] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11324.388856] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11324.390429] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11324.410795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11324.418288] Lustre: Skipped 1 previous similar message [11329.509969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11329.513028] Lustre: Skipped 2 previous similar messages [11330.665964] Lustre: server umount lustre-OST0000 complete [11335.657672] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11335.667469] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11335.678185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11339.767298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11339.776838] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11341.617536] Lustre: server umount lustre-MDT0000 complete [11345.634302] LustreError: 301944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787406387 with bad export cookie 3880959162359413264 [11345.636406] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11345.645475] LustreError: 301944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11346.066857] Lustre: server umount lustre-MDT0001 complete [11358.054722] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [11360.708767] Key type lgssc unregistered [11361.004876] LNet: 315220:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11361.014519] LNetError: 315220:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11361.035249] LNet: Removed LNI 192.168.201.156@tcp [11362.073163] Key type .llcrypt unregistered [11362.077153] Key type ._llcrypt unregistered [11396.583123] Key type ._llcrypt registered [11396.586859] Key type .llcrypt registered [11396.723761] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [11415.084798] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [11416.227548] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11416.326474] alg: No test for adler32 (adler32-zlib) [11417.807793] Lustre: Lustre: Build Version: 2.17.54_108_g575fb00 [11418.234635] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [11419.978525] Key type lgssc registered [11421.513720] Lustre: Echo OBD driver; http://www.lustre.org/ [11459.455828] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11459.473095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11460.793079] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11460.862919] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11460.970850] Lustre: lustre-MDT0000: new disk, initializing [11461.099865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11461.161464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11466.164993] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [11477.890890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11478.108273] Lustre: 319080:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11478.190545] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11478.194161] Lustre: Skipped 1 previous similar message [11478.327148] Lustre: lustre-MDT0001: new disk, initializing [11478.514448] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11478.567264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11478.581660] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11484.413143] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [11495.309182] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11500.971381] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11509.392955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11509.850869] Lustre: lustre-OST0000: new disk, initializing [11509.854969] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11509.862455] Lustre: 321048:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11509.977852] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11516.292540] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [11518.067303] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11518.081496] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11518.203684] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11526.034968] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11528.061638] Lustre: Mounted lustre-client [11528.858872] Lustre: Unmounted lustre-client [11538.410421] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11538.413205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11538.433130] Lustre: Skipped 1 previous similar message [11538.446821] Lustre: Skipped 1 previous similar message [11543.545208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11543.555122] Lustre: Skipped 1 previous similar message [11548.646515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11548.664533] Lustre: Skipped 1 previous similar message [11552.223490] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11552.516886] Lustre: server umount lustre-OST0000 complete [11558.883312] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11558.890249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11558.908411] Lustre: Skipped 1 previous similar message [11558.918560] Lustre: Skipped 1 previous similar message [11564.002977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11564.444421] Lustre: server umount lustre-MDT0000 complete [11569.789110] LustreError: 319070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787406612 with bad export cookie 13178439901685755315 [11569.790366] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11569.812137] LustreError: 319070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11570.185783] Lustre: server umount lustre-MDT0001 complete [11582.252915] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [11585.618561] Key type lgssc unregistered [11587.359169] LNet: 1 local NIs in recovery (showing 1): 192.168.201.156@tcp [11588.255987] LNet: 323210:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11588.280762] LNetError: 323210:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11658.985155] LNet: Removed LNI 192.168.201.156@tcp [11660.228338] Key type .llcrypt unregistered [11660.230509] Key type ._llcrypt unregistered [11679.290827] Key type ._llcrypt registered [11679.293615] Key type .llcrypt registered [11679.425311] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [11697.179688] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [11698.800108] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11699.197202] alg: No test for adler32 (adler32-zlib) [11700.380653] Lustre: Lustre: Build Version: 2.17.54_108_g575fb00 [11700.866369] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [11702.671257] Key type lgssc registered [11704.408534] Lustre: Echo OBD driver; http://www.lustre.org/ [11740.135241] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 09:53:00 (1787406780) [11748.770599] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11748.792793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11750.108255] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11750.139961] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11750.219396] Lustre: lustre-MDT0000: new disk, initializing [11750.356317] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11750.389285] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11756.148206] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [11768.517447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11768.720487] Lustre: 326434:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11768.820556] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11768.830302] Lustre: Skipped 1 previous similar message [11768.950337] Lustre: lustre-MDT0001: new disk, initializing [11769.131356] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11769.194518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11769.205331] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11774.098864] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [11785.051115] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11792.148829] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11800.126145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11800.615822] Lustre: lustre-OST0000: new disk, initializing [11800.624730] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11800.637858] Lustre: 328406:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11800.841655] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11809.603816] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [11810.430246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11810.442722] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11810.624976] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11819.869697] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11841.504129] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11841.510553] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11841.510967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11841.566146] LustreError: Skipped 1 previous similar message [11845.903080] Lustre: server umount lustre-OST0000 complete [11851.232224] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11851.242105] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11851.257616] Lustre: Skipped 1 previous similar message [11851.261358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11851.264204] Lustre: Skipped 1 previous similar message [11856.870570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11856.892624] Lustre: Skipped 2 previous similar messages [11857.508257] Lustre: server umount lustre-MDT0000 complete [11862.843932] LustreError: 326426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787406905 with bad export cookie 2211949097495162715 [11862.845643] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11862.855642] LustreError: 326426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11863.409348] Lustre: server umount lustre-MDT0001 complete [11875.269964] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [11877.928743] Key type lgssc unregistered [11878.208967] LNet: 330319:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11878.215660] LNetError: 330319:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11879.279138] LNet: Removed LNI 192.168.201.156@tcp [11880.309233] Key type .llcrypt unregistered [11880.316437] Key type ._llcrypt unregistered [11901.532780] Key type ._llcrypt registered [11901.535605] Key type .llcrypt registered [11901.629799] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [11902.613870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11902.863341] alg: No test for adler32 (adler32-zlib) [11904.181767] Lustre: Lustre: Build Version: 2.17.54_108_g575fb00 [11904.450971] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [11906.151227] Key type lgssc registered [11907.930602] Lustre: Echo OBD driver; http://www.lustre.org/ [11917.198911] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11917.221103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11918.657887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11924.312823] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [11928.032560] LustreError: 331595:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11928.046849] LustreError: 331595:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11933.155843] LustreError: 331596:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11933.681627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11934.116969] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11939.722397] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [11949.810993] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11956.940718] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11964.545694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11964.992737] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11971.757486] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [11980.569631] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12002.492941] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12004.532239] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12008.464119] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.56@tcp (stopping) [12009.453350] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12009.466800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12009.475751] Lustre: Skipped 2 previous similar messages [12009.485899] Lustre: Skipped 2 previous similar messages [12013.557284] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.56@tcp (stopping) [12014.758713] Lustre: server umount lustre-MDT0000 complete [12018.667907] LustreError: 332705:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12019.084100] LustreError: 331575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787407061 with bad export cookie 1303816168219146071 [12019.102920] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12019.683191] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12019.706106] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12019.716645] Lustre: Skipped 3 previous similar messages [12023.797611] LustreError: 331592:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12023.815212] LustreError: 331592:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12023.844928] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.56@tcp (stopping) [12025.528667] Lustre: server umount lustre-MDT0001 complete [12029.484852] Lustre: server umount lustre-OST0000 complete [12039.132708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12044.334516] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [12047.298157] Lustre: 335798:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.156@tcp [12049.854699] Lustre: server umount MGS complete [12061.030354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12061.888446] LustreError: 336371:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12061.908858] LustreError: 336371:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12062.074046] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12068.369813] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [12072.428952] LustreError: 336372:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12072.484140] LustreError: 336372:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [12081.743752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12081.946916] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12088.398025] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [12099.750446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12100.089813] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12108.437120] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [12116.823904] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12118.609985] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12124.766537] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12127.231342] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12132.845667] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12132.852913] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12132.897036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12132.910889] Lustre: Skipped 2 previous similar messages [12134.618959] Lustre: server umount lustre-OST0000 complete [12143.074341] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12143.098996] Lustre: Skipped 1 previous similar message [12146.839435] Lustre: server umount lustre-MDT0000 complete [12148.192455] LustreError: 336372:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12148.206086] LustreError: 336372:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [12151.152331] LustreError: 336351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787407193 with bad export cookie 1303816168219146911 [12151.154281] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12151.160760] LustreError: 336351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [12151.684641] Lustre: server umount lustre-MDT0001 complete [12164.504272] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [12167.118932] Key type lgssc unregistered [12167.451269] LNet: 339809:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12167.458444] LNetError: 339809:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12168.497495] LNet: Removed LNI 192.168.201.156@tcp [12169.838876] Key type .llcrypt unregistered [12169.842741] Key type ._llcrypt unregistered [12190.013415] Key type ._llcrypt registered [12190.015216] Key type .llcrypt registered [12190.140838] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [12205.622569] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [12206.713824] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12206.891947] alg: No test for adler32 (adler32-zlib) [12208.009503] Lustre: Lustre: Build Version: 2.17.54_108_g575fb00 [12208.244444] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [12209.927402] Key type lgssc registered [12210.952948] Lustre: Echo OBD driver; http://www.lustre.org/ [12248.406826] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12248.462627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12249.704947] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12249.733078] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12249.801542] Lustre: lustre-MDT0000: new disk, initializing [12249.883644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12249.913912] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12255.491834] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [12267.575428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12267.785291] Lustre: 342981:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12267.879602] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12267.888601] Lustre: Skipped 1 previous similar message [12268.006780] Lustre: lustre-MDT0001: new disk, initializing [12268.160106] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12268.233188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12268.246197] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12273.477724] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [12283.928718] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12291.213792] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12298.879144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12299.174644] Lustre: lustre-OST0000: new disk, initializing [12299.177995] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12299.184174] Lustre: 344948:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12299.347500] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12305.584618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12305.594858] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12305.673095] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12306.416716] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [12316.092437] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12321.586196] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12321.855435] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12326.571300] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12326.853081] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12331.488543] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12331.496062] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12331.514495] LustreError: Skipped 1 previous similar message [12331.518505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12331.529521] Lustre: Skipped 1 previous similar message [12331.563569] Lustre: Skipped 1 previous similar message [12335.466373] Lustre: server umount lustre-OST0000 complete [12339.683715] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12339.701492] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12339.724691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12341.730189] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12341.734854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12341.760601] Lustre: Skipped 1 previous similar message [12345.914818] Lustre: server umount lustre-MDT0000 complete [12350.123504] LustreError: 342976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787407392 with bad export cookie 14936087175247187281 [12350.125333] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12350.137531] LustreError: 342976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12350.435734] Lustre: server umount lustre-MDT0001 complete [12361.450766] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 10:03:21 (1787407401) [12384.454958] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [12398.845932] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [12439.885478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12440.270430] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12440.311571] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12440.433883] Lustre: lustre-MDT0000: new disk, initializing [12440.583803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12440.632436] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12445.686783] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [12457.854334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12458.044696] Lustre: 349725:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12458.201745] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12458.206586] Lustre: Skipped 1 previous similar message [12458.374308] Lustre: lustre-MDT0001: new disk, initializing [12458.538458] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12458.598266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12458.625700] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12464.246861] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [12475.805880] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12481.008733] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12487.880451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12488.198499] Lustre: lustre-OST0000: new disk, initializing [12488.203195] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12488.212470] Lustre: 351694:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12488.342582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12493.980862] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [12498.529959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12498.542238] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12498.678700] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12502.980250] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12534.766492] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12534.768875] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12534.769221] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12534.790280] Lustre: Skipped 1 previous similar message [12534.821569] LustreError: Skipped 1 previous similar message [12539.893599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12539.901740] Lustre: Skipped 2 previous similar messages [12540.241545] Lustre: server umount lustre-OST0000 complete [12544.992441] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12545.005529] Lustre: Skipped 1 previous similar message [12550.118086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12550.125127] LustreError: 349736:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12550.126955] Lustre: Skipped 2 previous similar messages [12550.161018] LustreError: 349736:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12550.271064] Lustre: server umount lustre-MDT0000 complete [12553.548326] LustreError: 349718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787407595 with bad export cookie 14936087175247188163 [12553.556315] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12553.562289] LustreError: 349718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12553.996798] Lustre: server umount lustre-MDT0001 complete [12565.191300] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [12568.495971] Key type lgssc unregistered [12568.805308] LNet: 353848:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12568.813828] LNetError: 353848:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12568.831287] LNet: Removed LNI 192.168.201.156@tcp [12569.542284] Key type .llcrypt unregistered [12569.544217] Key type ._llcrypt unregistered [12668.464622] Key type ._llcrypt registered [12668.468461] Key type .llcrypt registered [12668.591643] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [12668.973961] Key type .llcrypt unregistered [12668.977934] Key type ._llcrypt unregistered [12753.339846] Key type ._llcrypt registered [12753.341861] Key type .llcrypt registered [12753.453038] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [12753.718664] Key type .llcrypt unregistered [12753.725367] Key type ._llcrypt unregistered [12789.048537] Key type ._llcrypt registered [12789.050877] Key type .llcrypt registered [12789.121571] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [12789.423357] Key type .llcrypt unregistered [12789.427386] Key type ._llcrypt unregistered [12822.725338] Key type ._llcrypt registered [12822.727173] Key type .llcrypt registered [12822.802779] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [12823.020296] Key type .llcrypt unregistered [12823.022490] Key type ._llcrypt unregistered [12907.987674] Key type ._llcrypt registered [12907.990608] Key type .llcrypt registered [12908.085632] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [12908.306673] Key type .llcrypt unregistered [12908.309436] Key type ._llcrypt unregistered [12940.551796] Key type ._llcrypt registered [12940.554801] Key type .llcrypt registered [12940.678249] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [12941.075352] Key type .llcrypt unregistered [12941.077758] Key type ._llcrypt unregistered [12974.620500] Key type ._llcrypt registered [12974.623973] Key type .llcrypt registered [12974.728543] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [12975.138311] Key type .llcrypt unregistered [12975.141259] Key type ._llcrypt unregistered [13006.713788] Key type ._llcrypt registered [13006.738419] Key type .llcrypt registered [13006.884428] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [13007.102292] Key type .llcrypt unregistered [13007.104107] Key type ._llcrypt unregistered [13038.316741] Key type ._llcrypt registered [13038.319284] Key type .llcrypt registered [13038.417633] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [13038.660311] Key type .llcrypt unregistered [13038.662093] Key type ._llcrypt unregistered [13081.610827] Key type ._llcrypt registered [13081.614648] Key type .llcrypt registered [13081.771133] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [13082.150669] Key type .llcrypt unregistered [13082.157366] Key type ._llcrypt unregistered [13156.454272] Key type ._llcrypt registered [13156.456337] Key type .llcrypt registered [13156.598873] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [13156.924653] Key type .llcrypt unregistered [13156.926967] Key type ._llcrypt unregistered [13191.631750] Key type ._llcrypt registered [13191.634755] Key type .llcrypt registered [13191.821205] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [13192.136360] Key type .llcrypt unregistered [13192.138742] Key type ._llcrypt unregistered [13226.900480] Key type ._llcrypt registered [13226.903295] Key type .llcrypt registered [13227.035818] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [13227.378362] Key type .llcrypt unregistered [13227.380180] Key type ._llcrypt unregistered [13283.711926] Key type ._llcrypt registered [13283.720294] Key type .llcrypt registered [13283.835210] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [13299.777607] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [13301.442816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13301.541689] alg: No test for adler32 (adler32-zlib) [13302.842995] Lustre: Lustre: Build Version: 2.17.54_108_g575fb00 [13303.150266] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [13304.959228] Key type lgssc registered [13306.052425] Lustre: Echo OBD driver; http://www.lustre.org/ [13341.262770] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13341.315243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13342.651239] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13342.682093] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13342.742438] Lustre: tst250fs-MDT0000: new disk, initializing [13342.792981] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13342.810429] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13348.278665] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [13359.934526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13360.139170] Lustre: 374214:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13360.245718] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13360.252862] Lustre: Skipped 1 previous similar message [13360.374418] Lustre: tst250fs-MDT0001: new disk, initializing [13360.487200] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13360.523656] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13360.532762] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13365.929881] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [13371.097465] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13377.304145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13377.545436] Lustre: tst250fs-OST0000: new disk, initializing [13377.554554] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13377.567431] Lustre: 375845:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13377.657800] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13381.136506] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13381.149556] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13381.337133] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13384.067176] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [13395.409379] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13395.562845] Lustre: tst250fs-OST0001: new disk, initializing [13395.568334] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13395.576232] Lustre: 376719:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13395.715415] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13402.867869] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [13403.386742] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13403.402669] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13403.575406] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13409.447202] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13411.496513] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [13411.504735] Lustre: Skipped 1 previous similar message [13434.341134] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13434.378328] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13434.378445] Lustre: Skipped 1 previous similar message [13434.410552] Lustre: Skipped 3 previous similar messages [13439.459009] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13439.485156] Lustre: Skipped 3 previous similar messages [13439.964523] Lustre: server umount tst250fs-MDT0000 complete [13444.095316] LustreError: 374207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787408486 with bad export cookie 991331218238164517 [13444.100793] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13444.105402] LustreError: 374207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13444.588216] Lustre: tst250fs-MDT0001-lwp-OST0000: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13444.601491] Lustre: Skipped 2 previous similar messages [13444.745773] Lustre: server umount tst250fs-MDT0001 complete [13450.276489] Lustre: server umount tst250fs-OST0000 complete [13454.286164] Lustre: server umount tst250fs-OST0001 complete [13476.592776] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [13487.283884] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [13518.758730] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 10:22:39 (1787408559) [13520.279677] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13538.093977] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [13547.592479] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [13579.022323] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13273 sec ========= 10:23:39 (1787408619) [13581.169608] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 10:23:41 (1787408621) === [13584.935353] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 10:23:45 (1787408625) === [13629.630417] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [13632.770524] Key type lgssc unregistered [13633.028345] LNet: 385846:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13633.041193] LNetError: 385846:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13633.064770] LNet: Removed LNI 192.168.201.156@tcp [13634.396171] Key type .llcrypt unregistered [13634.397849] Key type ._llcrypt unregistered