[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 465069462 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002380] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004009] kvm-guest: setup PV IPIs [ 0.007501] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.008034] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.009016] pid_max: default: 32768 minimum: 301 [ 0.010184] LSM: Security Framework initializing [ 0.012011] Yama: becoming mindful. [ 0.013000] SELinux: Initializing. [ 0.013093] *** VALIDATE selinux *** [ 0.020699] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025431] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026185] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028071] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029125] *** VALIDATE tmpfs *** [ 0.030411] *** VALIDATE proc *** [ 0.031178] *** VALIDATE cgroup *** [ 0.032005] *** VALIDATE cgroup2 *** [ 0.034261] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035153] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037033] Spectre V2 : User space: Vulnerable [ 0.038007] Speculative Store Bypass: Vulnerable [ 0.041062] debug: unmapping init [mem 0xffffffffa5259000-0xffffffffa5260fff] [ 0.044196] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045826] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046029] ... version: 2 [ 0.047014] ... bit width: 48 [ 0.048012] ... generic registers: 4 [ 0.049013] ... value mask: 0000ffffffffffff [ 0.050011] ... max period: 00007fffffffffff [ 0.051012] ... fixed-purpose events: 3 [ 0.052010] ... event mask: 000000070000000f [ 0.053367] rcu: Hierarchical SRCU implementation. [ 0.055359] smp: Bringing up secondary CPUs ... [ 0.056616] x86: Booting SMP configuration: [ 0.057022] .... node #0, CPUs: #1 #2 #3 [ 0.061492] smp: Brought up 1 node, 4 CPUs [ 0.063020] smpboot: Max logical packages: 1 [ 0.064015] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.162072] node 0 deferred pages initialised in 97ms [ 0.166021] devtmpfs: initialized [ 0.167194] x86/mm: Memory block size: 128MB [ 0.169613] gcov: version magic: 0x41383552 [ 0.171173] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.174123] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.178433] pinctrl core: initialized pinctrl subsystem [ 0.180195] [ 0.180783] ************************************************************* [ 0.183017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.185012] ** ** [ 0.187014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.189012] ** ** [ 0.192013] ** This means that this kernel is built to expose internal ** [ 0.194012] ** IOMMU data structures, which may compromise security on ** [ 0.197016] ** your system. ** [ 0.199012] ** ** [ 0.201019] ** If you see this message and you are not debugging the ** [ 0.203015] ** kernel, report this immediately to your vendor! ** [ 0.206012] ** ** [ 0.208010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.210015] ************************************************************* [ 0.212716] NET: Registered protocol family 16 [ 0.215476] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.217064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.220079] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.224046] cpuidle: using governor menu [ 0.226024] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.228653] PCI: Using configuration type 1 for base access [ 0.231136] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.238133] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.241040] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.244122] cryptd: max_cpu_qlen set to 1000 [ 0.246355] ACPI: Added _OSI(Module Device) [ 0.248013] ACPI: Added _OSI(Processor Device) [ 0.249012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.251012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.256144] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.263228] ACPI: Interpreter enabled [ 0.264066] ACPI: PM: (supports S0 S3 S4 S5) [ 0.267020] ACPI: Using IOAPIC for interrupt routing [ 0.269135] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.272475] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.281883] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.284086] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.288030] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.294140] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.300088] acpiphp: Slot [2] registered [ 0.302211] acpiphp: Slot [5] registered [ 0.304164] acpiphp: Slot [6] registered [ 0.306261] acpiphp: Slot [7] registered [ 0.307123] acpiphp: Slot [8] registered [ 0.309165] acpiphp: Slot [9] registered [ 0.311144] acpiphp: Slot [10] registered [ 0.312142] acpiphp: Slot [3] registered [ 0.314127] acpiphp: Slot [4] registered [ 0.316132] acpiphp: Slot [11] registered [ 0.317093] acpiphp: Slot [12] registered [ 0.319079] acpiphp: Slot [13] registered [ 0.320082] acpiphp: Slot [14] registered [ 0.321104] acpiphp: Slot [15] registered [ 0.323101] acpiphp: Slot [16] registered [ 0.325113] acpiphp: Slot [17] registered [ 0.326055] acpiphp: Slot [18] registered [ 0.328114] acpiphp: Slot [19] registered [ 0.329118] acpiphp: Slot [20] registered [ 0.331084] acpiphp: Slot [21] registered [ 0.333129] acpiphp: Slot [22] registered [ 0.335134] acpiphp: Slot [23] registered [ 0.337130] acpiphp: Slot [24] registered [ 0.338129] acpiphp: Slot [25] registered [ 0.340086] acpiphp: Slot [26] registered [ 0.341102] acpiphp: Slot [27] registered [ 0.343115] acpiphp: Slot [28] registered [ 0.344084] acpiphp: Slot [29] registered [ 0.346130] acpiphp: Slot [30] registered [ 0.347126] acpiphp: Slot [31] registered [ 0.349072] PCI host bridge to bus 0000:00 [ 0.351022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.354029] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.356027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.360045] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.362031] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.366015] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.368210] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.372000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.375427] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.384730] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.391040] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.393014] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.395025] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.397019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.400633] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.403788] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.406049] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.409016] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.415015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.428017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.434015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.440071] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.447020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.453020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.470020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.480296] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.485015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.492019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.506019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.518240] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.526019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.532016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.548014] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.563970] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.568018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.575020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.592015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.603928] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.612017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.618019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.633021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.644446] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.654020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.660021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.673033] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.682000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.686335] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.687348] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.690469] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.693266] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.698181] iommu: Default domain type: Passthrough [ 0.700557] SCSI subsystem initialized [ 0.702159] ACPI: bus type USB registered [ 0.703112] usbcore: registered new interface driver usbfs [ 0.705098] usbcore: registered new interface driver hub [ 0.708205] usbcore: registered new device driver usb [ 0.710189] pps_core: LinuxPPS API ver. 1 registered [ 0.712014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.716084] PTP clock support registered [ 0.718200] EDAC MC: Ver: 3.0.0 [ 0.721022] PCI: Using ACPI for IRQ routing [ 0.724088] NetLabel: Initializing [ 0.725010] NetLabel: domain hash size = 128 [ 0.726014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.728103] NetLabel: unlabeled traffic allowed by default [ 0.731132] vgaarb: loaded [ 0.733251] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.735011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.742722] clocksource: Switched to clocksource kvm-clock [ 0.841977] VFS: Disk quotas dquot_6.6.0 [ 0.842880] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.844749] *** VALIDATE ramfs *** [ 0.846159] *** VALIDATE hugetlbfs *** [ 0.847239] pnp: PnP ACPI init [ 0.850364] pnp: PnP ACPI: found 6 devices [ 0.865399] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.868807] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.871363] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.874020] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.876622] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.878946] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.881495] NET: Registered protocol family 2 [ 0.883324] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.887358] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.890235] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.895514] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.898674] TCP: Hash tables configured (established 65536 bind 65536) [ 0.902819] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.905984] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.908601] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.911606] NET: Registered protocol family 1 [ 0.914341] RPC: Registered named UNIX socket transport module. [ 0.916789] RPC: Registered udp transport module. [ 0.918683] RPC: Registered tcp transport module. [ 0.920593] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.923350] NET: Registered protocol family 44 [ 0.925092] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.927021] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.930257] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.932957] PCI: CLS 0 bytes, default 64 [ 0.935365] Unpacking initramfs... [ 2.289107] debug: unmapping init [mem 0xffff933d7cc54000-0xffff933d7ffbffff] [ 2.295578] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.298059] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.302206] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.753744] Initialise system trusted keyrings [ 2.755464] Key type blacklist registered [ 2.757319] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.765246] zbud: loaded [ 2.767503] *** VALIDATE nfs *** [ 2.768851] *** VALIDATE nfs4 *** [ 2.770022] pstore: using deflate compression [ 2.772337] Platform Keyring initialized [ 2.867179] NET: Registered protocol family 38 [ 2.870536] Key type asymmetric registered [ 2.872308] Asymmetric key parser 'x509' registered [ 2.874311] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.877643] io scheduler mq-deadline registered [ 2.879817] io scheduler kyber registered [ 2.881845] io scheduler bfq registered [ 2.883794] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.887381] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.890156] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.892938] ACPI: Power Button [PWRF] [ 2.898272] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.905323] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.920127] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.927484] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.943337] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.971488] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.999390] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.004312] Non-volatile memory driver v1.3 [ 3.006254] Linux agpgart interface v0.103 [ 3.038115] virtio_blk virtio1: [vda] 144824 512-byte logical blocks (74.1 MB/70.7 MiB) [ 3.041355] vda: detected capacity change from 0 to 74149888 [ 3.057712] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.061476] vdb: detected capacity change from 0 to 1073741824 [ 3.075279] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.077244] vdc: detected capacity change from 0 to 2621440000 [ 3.088246] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.090734] vdd: detected capacity change from 0 to 2621440000 [ 3.104027] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.107185] vde: detected capacity change from 0 to 4294967296 [ 3.120992] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.124397] vdf: detected capacity change from 0 to 4294967296 [ 3.129513] libphy: Fixed MDIO Bus: probed [ 3.135337] usbcore: registered new interface driver usbserial_generic [ 3.137855] usbserial: USB Serial support registered for generic [ 3.139964] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.145218] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.147196] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.149740] mousedev: PS/2 mouse device common for all mice [ 3.152248] rtc_cmos 00:05: RTC can wake from S4 [ 3.153084] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.155824] rtc_cmos 00:05: registered as rtc0 [ 3.160978] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.161188] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.161228] intel_pstate: CPU model not supported [ 3.165456] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.170676] hid: raw HID events driver (C) Jiri Kosina [ 3.173383] usbcore: registered new interface driver usbhid [ 3.175566] usbhid: USB HID core driver [ 3.176934] drop_monitor: Initializing network drop monitor service [ 3.180203] Initializing XFRM netlink socket [ 3.181748] NET: Registered protocol family 10 [ 3.184511] Segment Routing with IPv6 [ 3.186268] NET: Registered protocol family 17 [ 3.188200] mpls_gso: MPLS GSO support [ 3.193868] RAS: Correctable Errors collector initialized. [ 3.196152] AVX version of gcm_enc/dec engaged. [ 3.197796] AES CTR mode by8 optimization enabled [ 3.283926] sched_clock: Marking stable (3283891689, 0)->(4109622187, -825730498) [ 3.289604] registered taskstats version 1 [ 3.295322] Loading compiled-in X.509 certificates [ 3.297173] zswap: loaded using pool lzo/zbud [ 3.319067] Key type big_key registered [ 3.330659] Key type encrypted registered [ 3.332228] ima: No TPM chip found, activating TPM-bypass! [ 3.333791] ima: Allocated hash algorithm: sha1 [ 3.335183] ima: No architecture policies found [ 3.336855] evm: Initialising EVM extended attributes: [ 3.338435] evm: security.selinux [ 3.340010] evm: security.ima [ 3.341248] evm: security.capability [ 3.342946] evm: HMAC attrs: 0x1 [ 3.345122] rtc_cmos 00:05: setting system clock to 2026-06-28 04:48:42 UTC (1782622122) [ 3.352099] debug: unmapping init [mem 0xffffffffa6203000-0xffffffffa63fffff] [ 3.355793] debug: unmapping init [mem 0xffffffffa4f82000-0xffffffffa5258fff] [ 3.365095] Write protecting the kernel read-only data: 28672k [ 3.369253] debug: unmapping init [mem 0xffffffffa3603000-0xffffffffa37fffff] [ 3.371805] debug: unmapping init [mem 0xffffffffa3f14000-0xffffffffa3ffffff] [ 3.406641] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.416166] systemd[1]: Detected virtualization kvm. [ 3.417684] systemd[1]: Detected architecture x86-64. [ 3.419607] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.442987] systemd[1]: No hostname configured. [ 3.444945] systemd[1]: Set hostname to . [ 3.447491] random: systemd: uninitialized urandom read (16 bytes read) [ 3.450386] systemd[1]: Initializing machine ID from random generator. [ 3.497575] random: ln: uninitialized urandom read (6 bytes read) [ 3.588677] random: systemd: uninitialized urandom read (16 bytes read) [ 3.591273] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.596409] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.600356] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.211683] device-mapper: uevent: version 1.0.3 [ 4.214220] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.877944] virtio_net virtio0 ens2: renamed from eth0 [ 4.884443] random: fast init done [ 4.890726] scsi host0: ata_piix [ 4.902191] scsi host1: ata_piix [ 4.903661] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.909150] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.697452] dracut-initqueue[581]: RTNETLINK answers: File exists [ 9.847197] random: crng init done [ 9.848860] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.179680] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.188350] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.394256] SELinux: Disabled at runtime. [ 11.454960] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.464737] systemd[1]: Detected virtualization kvm. [ 11.466677] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.890546] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.893561] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.898228] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.902462] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.906908] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.914662] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.919568] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. Mounting Kernel Debug File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. Mounting POSIX Message Queue File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-serial\x2dgetty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ 12.019201] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Slices. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-getty.slice. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.310610] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.639780] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.654900] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.766096] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.776315] EDAC sbridge: Ver: 1.1.2 [ 14.013162] Key type dns_resolver registered [ 14.305508] NFS: Registering the id_resolver key type [ 14.308090] Key type id_resolver registered [ 14.309716] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg409-server login: [ 37.846903] libcfs: loading out-of-tree module taints kernel. [ 37.867395] Key type ._llcrypt registered [ 37.868822] Key type .llcrypt registered [ 37.916328] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [ 46.313706] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 47.028171] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 47.035296] alg: No test for adler32 (adler32-zlib) [ 48.114299] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 48.486635] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 50.120177] Key type lgssc registered [ 50.914844] Lustre: Echo OBD driver; http://www.lustre.org/ [ 58.597524] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 75.132444] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 81.027784] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 81.047423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 82.169920] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 82.185459] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 82.227253] Lustre: lustre-MDT0000: new disk, initializing [ 82.262485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 82.271933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 84.035719] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 90.141088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 90.199325] Lustre: 6496:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 90.223662] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 90.227387] Lustre: Skipped 1 previous similar message [ 90.271828] Lustre: lustre-MDT0001: new disk, initializing [ 90.307793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 90.332732] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 90.340974] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 92.053711] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 94.603163] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 98.652972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 98.768876] Lustre: lustre-OST0000: new disk, initializing [ 98.770736] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 98.774237] Lustre: 8432:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 98.802180] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 100.278110] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 100.282147] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 100.318116] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 101.184680] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 107.813616] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 107.863849] Lustre: lustre-OST0001: new disk, initializing [ 107.865877] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 107.868933] Lustre: 9509:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 107.894416] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 108.315091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 108.318256] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 108.353092] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 110.154917] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 116.690644] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 122.612663] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 125.103874] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing check_logdir /tmp/testlogs/ [ 126.900990] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing yml_node [ 128.544044] Lustre: DEBUG MARKER: Client: 2.17.54.105 [ 129.520128] Lustre: DEBUG MARKER: MDS: 2.17.54.105 [ 130.507495] Lustre: DEBUG MARKER: OSS: 2.17.54.105 [ 131.144183] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Jun 28 00:50:50 EDT 2026 [ 137.429820] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 137.979908] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 152.032457] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 152.036709] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 152.042534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 154.592839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 154.593486] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 154.596233] Lustre: Skipped 2 previous similar messages [ 154.601189] Lustre: Skipped 2 previous similar messages [ 159.713060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 159.715531] Lustre: Skipped 4 previous similar messages [ 164.320120] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 164.395246] Lustre: server umount lustre-MDT0000 complete [ 165.757647] LustreError: 7444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782622284 with bad export cookie 6415298647139488031 [ 165.759203] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 165.762593] LustreError: 7444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 169.952753] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 169.952963] LustreError: 6505:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 169.953086] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 169.961702] Lustre: Skipped 1 previous similar message [ 169.969117] LustreError: 6505:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 170.977881] LustreError: 12137:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 170.989850] LustreError: 12137:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 175.072958] LustreError: 6506:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 175.072960] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 175.081752] Lustre: Skipped 4 previous similar messages [ 180.192223] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 180.192777] LustreError: 8630:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 180.202368] LustreError: 8630:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 180.275273] Lustre: server umount lustre-MDT0001 complete [ 187.729631] Lustre: server umount lustre-OST0000 complete [ 195.351940] Lustre: server umount lustre-OST0001 complete [ 197.764768] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [ 201.653442] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 215.656608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 215.777194] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 215.789822] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 215.837440] Lustre: lustre-MDT0000: new disk, initializing [ 215.867447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 215.875756] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 217.317766] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 222.067808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 222.130753] Lustre: 15052:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 222.251139] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 223.890035] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 227.863722] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 229.841792] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 232.626473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 232.761925] Lustre: lustre-OST0000: new disk, initializing [ 232.764586] Lustre: Skipped 1 previous similar message [ 232.766731] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 232.769211] Lustre: Skipped 2 previous similar messages [ 232.773732] Lustre: 17025:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 232.823269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 232.826642] Lustre: Skipped 1 previous similar message [ 234.575600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 234.580059] Lustre: Skipped 1 previous similar message [ 234.583845] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 234.624240] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 235.362755] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 239.511479] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 241.035040] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 241.112406] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 242.877373] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 242.959976] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 245.218834] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 245.220131] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 245.220714] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 245.220718] Lustre: Skipped 2 previous similar messages [ 245.221454] LustreError: Skipped 1 previous similar message [ 245.232684] Lustre: Skipped 1 previous similar message [ 249.824300] Lustre: server umount lustre-OST0000 complete [ 252.897243] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 252.900570] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 257.578984] Lustre: server umount lustre-MDT0000 complete [ 259.220570] LustreError: 15047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782622378 with bad export cookie 6415298647139489557 [ 259.221682] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 259.226839] LustreError: 15047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 259.374107] Lustre: server umount lustre-MDT0001 complete [ 261.876889] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 00:53:01 (1782622381) [ 269.217733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 269.430932] LustreError: 19896:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 269.464058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 271.090921] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 274.852482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 276.621986] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 279.611525] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 281.775591] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 282.728603] Lustre: Failing over lustre-MDT0000 [ 282.830743] Lustre: server umount lustre-MDT0000 complete [ 284.641433] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 284.642700] LustreError: 19891:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 284.648282] Lustre: Skipped 2 previous similar messages [ 284.655421] LustreError: 19891:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 296.724025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 296.804752] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 298.618927] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 300.004041] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 300.007610] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 300.022499] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 303.218761] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 310.241771] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 310.241943] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 310.249744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 310.256403] Lustre: Skipped 5 previous similar messages [ 311.657705] Lustre: server umount lustre-MDT0000 complete [ 313.285384] LustreError: 19876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782622432 with bad export cookie 6415298647139490971 [ 313.288730] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 313.290031] LustreError: 19876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 313.416574] Lustre: server umount lustre-MDT0001 complete [ 317.189422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 317.414751] LustreError: 23227:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 317.422191] LustreError: 23227:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 317.448747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 317.453893] Lustre: Skipped 2 previous similar messages [ 319.105631] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 322.635245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 324.359347] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 327.260242] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 329.486428] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 330.350547] Lustre: Failing over lustre-MDT0000 [ 330.447814] Lustre: server umount lustre-MDT0000 complete [ 343.800338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 343.852623] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 345.468422] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 348.070448] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 348.073038] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 348.077643] Lustre: Skipped 1 previous similar message [ 348.083471] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 349.470287] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 353.249896] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 353.250629] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 353.261469] Lustre: Skipped 3 previous similar messages [ 353.264294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 353.267301] Lustre: Skipped 1 previous similar message [ 357.655416] Lustre: server umount lustre-MDT0000 complete [ 358.369905] LustreError: 23966:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 358.374855] LustreError: 23966:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 358.906554] LustreError: 23210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782622478 with bad export cookie 6415298647139491909 [ 358.909314] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 358.910061] LustreError: 23210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 359.027148] Lustre: server umount lustre-MDT0001 complete [ 361.892221] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 00:54:41 (1782622481) [ 367.544461] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [ 370.874637] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 384.569562] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 388.244423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 388.326125] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 388.336734] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 388.372150] Lustre: lustre-MDT0000: new disk, initializing [ 388.391931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 388.394961] Lustre: Skipped 2 previous similar messages [ 388.400498] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 389.785468] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 394.113192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 394.155849] Lustre: 29589:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 394.271215] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 395.608887] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 399.291758] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 401.050322] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 403.308662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 403.413322] Lustre: 31558:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 404.805526] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 404.827628] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 404.834450] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 405.356105] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 409.144533] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 411.274216] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 417.961317] Lustre: server umount lustre-OST0000 complete [ 419.809109] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 419.812045] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 419.816479] Lustre: Skipped 1 previous similar message [ 419.819160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 429.565886] Lustre: server umount lustre-MDT0000 complete [ 430.561763] LustreError: 29598:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 430.568223] LustreError: 29598:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 430.892370] LustreError: 30383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782622550 with bad export cookie 6415298647139492315 [ 430.896459] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 430.898169] LustreError: 30383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 441.382618] Lustre: server umount lustre-MDT0001 complete [ 445.017832] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 446.031842] Key type lgssc unregistered [ 446.157330] LNet: 33511:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 446.160343] LNetError: 33511:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 446.167332] LNet: Removed LNI 192.168.204.109@tcp [ 446.509129] Key type .llcrypt unregistered [ 446.510738] Key type ._llcrypt unregistered [ 455.711908] Key type ._llcrypt registered [ 455.713318] Key type .llcrypt registered [ 455.750386] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 456.262368] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 456.276613] alg: No test for adler32 (adler32-zlib) [ 457.128810] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 457.218100] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 458.800126] Key type lgssc registered [ 459.152467] Lustre: Echo OBD driver; http://www.lustre.org/ [ 461.923388] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 461.931889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 463.092988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 464.621992] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 467.932665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 468.062243] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 469.445504] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 471.871353] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 473.645819] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 475.832602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 475.947305] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 477.880283] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 477.991358] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 480.167446] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 495.252196] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 00:56:54 (1782622614) [ 495.875966] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 496.477358] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 498.547255] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 00:56:57 (1782622617) [ 501.512789] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 00:57:00 (1782622620) [ 501.854902] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 504.848281] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 504.850709] Lustre: Skipped 2 previous similar messages [ 505.408546] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 00:57:04 (1782622624) [ 512.594027] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 512.596778] Lustre: Skipped 10 previous similar messages [ 513.487937] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 00:57:12 (1782622632) [ 514.449729] Lustre: *** cfs_fail_loc=131b, val=2*** [ 515.696331] Lustre: *** cfs_fail_loc=131b, val=2*** [ 518.511794] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 00:57:17 (1782622637) [ 520.613834] Lustre: Disabling parameter general.jobid_name=TESTNAME1 in log params [ 520.616202] Lustre: Skipped 6 previous similar messages [ 523.342487] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 00:57:22 (1782622642) [ 529.377796] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 529.380198] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 529.388483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 529.391250] Lustre: Skipped 1 previous similar message [ 534.228708] Lustre: server umount lustre-OST0000 complete [ 539.617798] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 539.617929] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 539.618661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 539.620561] LustreError: Skipped 1 previous similar message [ 539.629883] Lustre: Skipped 2 previous similar messages [ 541.685248] Lustre: server umount lustre-MDT0000 complete [ 543.340023] LustreError: 36741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782622662 with bad export cookie 17791642743180018403 [ 543.343166] LustreError: 36741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 543.343625] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 543.461223] Lustre: server umount lustre-MDT0001 complete [ 547.378506] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 548.504350] Key type lgssc unregistered [ 548.630408] LNet: 42742:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 548.633047] LNetError: 42742:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 548.646427] LNet: Removed LNI 192.168.204.109@tcp [ 549.001091] Key type .llcrypt unregistered [ 549.002630] Key type ._llcrypt unregistered [ 555.467141] Key type ._llcrypt registered [ 555.468517] Key type .llcrypt registered [ 555.508858] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [ 562.441375] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 562.881337] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 562.888053] alg: No test for adler32 (adler32-zlib) [ 563.747997] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 563.839940] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 565.424175] Key type lgssc registered [ 565.836099] Lustre: Echo OBD driver; http://www.lustre.org/ [ 578.669930] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 578.676361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 579.759727] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 579.775794] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 579.814437] Lustre: lustre-MDT0000: new disk, initializing [ 579.839928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 579.847468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 581.243272] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 585.338335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 585.381403] Lustre: 45867:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 585.406187] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 585.407895] Lustre: Skipped 1 previous similar message [ 585.442426] Lustre: lustre-MDT0001: new disk, initializing [ 585.474754] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 585.487145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 585.492479] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 586.747918] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 590.384180] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 592.144618] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 594.429520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 594.524950] Lustre: lustre-OST0000: new disk, initializing [ 594.527109] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 594.530741] Lustre: 47834:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 594.557950] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 596.431534] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 600.151526] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 601.468141] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 602.102520] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 602.106709] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 602.135797] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 602.546554] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 603.903715] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 603.979229] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 607.204962] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 607.205230] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 607.211417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 607.213386] LustreError: Skipped 1 previous similar message [ 610.700553] Lustre: server umount lustre-OST0000 complete [ 612.321615] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 612.322253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 612.328458] Lustre: Skipped 2 previous similar messages [ 612.332564] Lustre: Skipped 2 previous similar messages [ 617.441235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 617.443551] Lustre: Skipped 1 previous similar message [ 622.561498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 622.564111] Lustre: Skipped 1 previous similar message [ 626.144166] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 626.204021] Lustre: server umount lustre-MDT0000 complete [ 627.446231] LustreError: 46662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782622746 with bad export cookie 1466201498295846955 [ 627.449468] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 627.450955] LustreError: 46662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 627.565697] Lustre: server umount lustre-MDT0001 complete [ 630.695853] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 00:59:10 (1782622750) [ 632.871685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 633.024233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 634.288338] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 637.047020] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 637.168176] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 638.537645] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 640.888799] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 642.760592] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 645.087058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 645.206950] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 647.181370] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 649.743684] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 662.072729] LustreError: 52845:0:(llog.c:444:llog_init_handle()) llog type is not specified! [ 662.434652] Lustre: Modifying parameter general.timeout=1 in log params [ 663.478495] Lustre: Modifying parameter general.timeout=4 in log params [ 663.480734] Lustre: Skipped 2 previous similar messages [ 665.507603] Lustre: Modifying parameter general.timeout=10 in log params [ 665.509826] Lustre: Skipped 5 previous similar messages [ 669.830834] Lustre: Modifying parameter general.timeout=22 in log params [ 669.833445] Lustre: Skipped 11 previous similar messages [ 677.886348] Lustre: Modifying parameter general.timeout=47 in log params [ 677.888862] Lustre: Skipped 24 previous similar messages [ 694.114229] Lustre: Modifying parameter general.timeout=99 in log params [ 694.116667] Lustre: Skipped 51 previous similar messages [ 706.630647] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 01:00:26 (1782622826) [ 707.581625] Lustre: 59747:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 708.831593] Lustre: 59943:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 708.839974] Lustre: 59943:0:(mgs_llog.c:1447:mgs_modify_param()) Skipped 1 previous similar message [ 719.386836] Lustre: 61589:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 719.389772] Lustre: 61589:0:(mgs_llog.c:1447:mgs_modify_param()) Skipped 3 previous similar messages [ 719.392249] LustreError: 61589:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 720.764445] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 01:00:40 (1782622840) [ 725.473681] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 725.475692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 725.479285] Lustre: Skipped 2 previous similar messages [ 725.483612] Lustre: Skipped 2 previous similar messages [ 726.748672] Lustre: server umount lustre-MDT0000 complete [ 728.118829] LustreError: 50276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782622847 with bad export cookie 1466201498295847823 [ 728.121570] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 728.123545] LustreError: 50276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 730.281622] Lustre: server umount lustre-MDT0001 complete [ 750.091913] Lustre: server umount lustre-OST0000 complete [ 759.484623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 759.584574] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 759.595849] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 759.600476] Lustre: Skipped 53 previous similar messages [ 759.687796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 761.031100] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 763.867475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 763.903292] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 763.913300] Lustre: 64126:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 765.422725] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 767.716947] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 769.466947] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 771.729637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 771.800273] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 773.739232] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 776.015947] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 776.550750] Lustre: 66692:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 778.723212] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 778.723345] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 778.726357] LustreError: Skipped 1 previous similar message [ 778.732925] Lustre: Skipped 1 previous similar message [ 778.735171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 778.736837] Lustre: Skipped 1 previous similar message [ 780.305760] Lustre: server umount lustre-OST0000 complete [ 783.703990] Lustre: server umount lustre-MDT0000 complete [ 783.841316] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 783.841954] LustreError: 66654:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 783.847993] Lustre: Skipped 1 previous similar message [ 783.853299] LustreError: 66654:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 784.867086] LustreError: 64118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782622904 with bad export cookie 1466201498295850098 [ 784.870742] LustreError: 64118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 784.870814] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 784.974306] Lustre: server umount lustre-MDT0001 complete [ 788.420538] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 789.358476] Key type lgssc unregistered [ 789.477323] LNet: 67893:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 789.480985] LNetError: 67893:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 789.489286] LNet: Removed LNI 192.168.204.109@tcp [ 789.785588] Key type .llcrypt unregistered [ 789.787255] Key type ._llcrypt unregistered [ 798.705082] Key type ._llcrypt registered [ 798.706100] Key type .llcrypt registered [ 798.744251] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 799.056038] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 799.092355] alg: No test for adler32 (adler32-zlib) [ 799.949724] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 800.040581] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 801.616158] Key type lgssc registered [ 801.988601] Lustre: Echo OBD driver; http://www.lustre.org/ [ 804.588873] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 804.593886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 805.734442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 807.052488] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 809.830985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 810.018495] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 811.292543] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 813.718199] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 815.534441] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 817.962797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 818.187077] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 820.283296] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 822.683667] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 836.907494] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 840.673137] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 840.676677] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 840.682767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 841.697583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 841.698914] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 845.458875] Lustre: server umount lustre-MDT0000 complete [ 846.716063] LustreError: 69264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782622965 with bad export cookie 14023257478599114007 [ 846.719854] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 846.721067] LustreError: 69264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 846.835741] Lustre: server umount lustre-MDT0001 complete [ 858.391637] Lustre: server umount lustre-OST0000 complete [ 867.415336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 867.517914] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 867.528660] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 867.611744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 868.983126] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 871.742258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 871.777790] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 871.787889] Lustre: 74168:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 871.937390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 873.289487] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 875.639335] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 877.339097] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 879.629367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 879.713460] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 881.764183] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 884.006967] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 884.505036] Lustre: 76741:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 885.135465] Lustre: Modifying parameter general.jobid_var=disable in log params [ 887.265543] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 887.266507] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 887.268748] LustreError: Skipped 1 previous similar message [ 887.273138] Lustre: Skipped 1 previous similar message [ 887.273788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 887.277445] Lustre: Skipped 3 previous similar messages [ 888.466330] Lustre: server umount lustre-OST0000 complete [ 892.385708] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 892.385919] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 892.386390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 892.389285] Lustre: Skipped 2 previous similar messages [ 895.763538] Lustre: server umount lustre-MDT0000 complete [ 896.968388] LustreError: 76745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782623016 with bad export cookie 14023257478599114819 [ 896.971885] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 896.972065] LustreError: 76745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 897.085230] Lustre: server umount lustre-MDT0001 complete [ 900.427874] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 901.350770] Key type lgssc unregistered [ 901.474369] LNet: 77947:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 901.476830] LNetError: 77947:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 901.484259] LNet: Removed LNI 192.168.204.109@tcp [ 901.756321] Key type .llcrypt unregistered [ 901.758075] Key type ._llcrypt unregistered [ 910.237397] Key type ._llcrypt registered [ 910.238309] Key type .llcrypt registered [ 910.272303] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 910.567095] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 910.588049] alg: No test for adler32 (adler32-zlib) [ 911.426656] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 911.504471] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 913.072147] Key type lgssc registered [ 913.382577] Lustre: Echo OBD driver; http://www.lustre.org/ [ 915.736372] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 915.742126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 916.876060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 918.129675] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 920.777802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 920.957648] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 922.212443] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 924.441049] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 926.111534] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 928.294401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 928.472107] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 930.274910] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 932.541624] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 945.428770] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 946.028882] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 946.031019] Lustre: Skipped 1 previous similar message [ 947.168280] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 947.170652] Lustre: Skipped 3 previous similar messages [ 949.430628] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 949.433243] Lustre: Skipped 7 previous similar messages [ 953.535497] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 953.536956] Lustre: Skipped 13 previous similar messages [ 961.605843] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 961.608633] Lustre: Skipped 28 previous similar messages [ 976.360608] Lustre: 87423:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 977.160561] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 01:04:56 (1782623096) [ 977.626307] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 978.212929] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 01:04:57 (1782623097) [ 983.008768] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 983.011863] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 983.017079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 984.033985] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 984.035574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 984.036347] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 985.164339] Lustre: server umount lustre-OST0000 complete [ 987.617074] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 987.620520] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 987.624791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 987.626813] Lustre: Skipped 1 previous similar message [ 992.534607] Lustre: server umount lustre-MDT0000 complete [ 993.723271] LustreError: 81946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782623112 with bad export cookie 14607162673534669665 [ 993.725349] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 993.727330] LustreError: 81946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 993.841462] Lustre: server umount lustre-MDT0001 complete [ 997.195226] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 998.106138] Key type lgssc unregistered [ 998.231402] LNet: 88670:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 998.234969] LNetError: 88670:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 998.244278] LNet: Removed LNI 192.168.204.109@tcp [ 998.545050] Key type .llcrypt unregistered [ 998.545929] Key type ._llcrypt unregistered [ 999.961480] Key type ._llcrypt registered [ 999.963226] Key type .llcrypt registered [ 1000.002650] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1001.396078] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules [ 1001.707924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1001.741053] alg: No test for adler32 (adler32-zlib) [ 1002.589463] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 1002.676984] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 1004.256144] Key type lgssc registered [ 1004.626207] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1005.056244] LustreError: 89824:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 1012.845764] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 1016.046586] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1026.282755] LustreError: 89824:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1026.293464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1027.479099] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1028.902380] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 1031.790909] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 01:05:51 (1782623151) [ 1033.429442] LustreError: 90878:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1033.434985] LustreError: 90878:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1033.439604] LustreError: 90878:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 0, failed: rc = -5 [ 1033.559138] Lustre: server umount lustre-MDT0000 complete [ 1038.219511] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 1039.211207] Key type lgssc unregistered [ 1039.348386] LNet: 92369:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1039.350974] LNetError: 92369:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1039.358808] LNet: Removed LNI 192.168.204.109@tcp [ 1039.643909] Key type .llcrypt unregistered [ 1039.645586] Key type ._llcrypt unregistered [ 1046.759118] Key type ._llcrypt registered [ 1046.760957] Key type .llcrypt registered [ 1046.805551] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 1047.166890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1047.174774] alg: No test for adler32 (adler32-zlib) [ 1048.028615] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 1048.132465] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 1049.720222] Key type lgssc registered [ 1050.153324] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1052.771453] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1052.777631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1053.915063] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1055.198465] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 1057.940816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1058.145230] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1059.409347] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 1061.564654] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1063.233116] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1065.351899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1065.535754] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1067.364985] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 1069.643494] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1100.783477] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 01:07:00 (1782623220) [ 1101.393525] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1102.024186] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 01:07:01 (1782623221) [ 1104.352765] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1104.355022] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1104.359429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1104.866622] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1104.867039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1104.872042] Lustre: Skipped 1 previous similar message [ 1109.083607] Lustre: server umount lustre-MDT0000 complete [ 1109.986852] LustreError: 93563:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1109.992118] LustreError: 93563:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1110.285131] LustreError: 94301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782623229 with bad export cookie 2755979026729669776 [ 1110.288494] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1110.290728] LustreError: 94301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1110.392856] Lustre: server umount lustre-MDT0001 complete [ 1121.928548] Lustre: server umount lustre-OST0000 complete [ 1126.003127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1126.146868] LustreError: 97847:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1126.197506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1127.582680] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 1130.713188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1130.961216] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1132.490264] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 1135.013661] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1136.785071] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1141.134541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1141.205249] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1141.208589] LustreError: 97835:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1141.219478] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1141.222142] LustreError: 99929:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 1141.224432] LustreError: 99929:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 1141.226412] LustreError: 99929:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1141.250887] Lustre: server umount lustre-OST0000 complete [ 1141.252831] LustreError: 99929:0:(super25.c:185:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1144.096454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1144.160052] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1144.163197] LustreError: 97836:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1144.171840] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1144.175054] LustreError: 100393:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 1144.177461] LustreError: 100393:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 1144.179617] LustreError: 100393:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1144.197157] Lustre: server umount lustre-OST0000 complete [ 1144.198404] LustreError: 100393:0:(super25.c:185:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1147.316130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1147.376375] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1147.378064] Lustre: Found index 0 for lustre-OST0000, updating log [ 1147.380130] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1147.398895] Lustre: lustre-OST0000: new disk, initializing [ 1147.400513] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1147.402938] Lustre: 100922:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1147.488397] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1149.390375] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 1150.610870] Lustre: Failing over lustre-OST0000 [ 1150.653293] Lustre: server umount lustre-OST0000 complete [ 1154.978971] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1154.979615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1154.982925] Lustre: Skipped 1 previous similar message [ 1154.985795] Lustre: Skipped 2 previous similar messages [ 1160.161543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1160.163273] Lustre: Skipped 1 previous similar message [ 1165.280919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1165.282582] Lustre: Skipped 1 previous similar message [ 1166.304103] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1166.362162] Lustre: server umount lustre-MDT0000 complete [ 1167.583236] LustreError: 98590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782623286 with bad export cookie 2755979026729672506 [ 1167.586565] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1167.587462] LustreError: 98590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1167.692392] Lustre: server umount lustre-MDT0001 complete [ 1170.525654] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 01:08:09 (1782623289) [ 1175.747876] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 1179.212116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1179.338438] LustreError: 103262:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1179.366575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1180.722927] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 1183.668243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1185.299552] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 1186.316913] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 1188.421931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1190.467725] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 1193.335145] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1193.369151] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 1193.400883] Lustre: lustre-OST0001: new disk, initializing [ 1193.403185] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1193.405366] Lustre: 105578:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1194.572941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1194.577776] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1194.611507] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1195.400165] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 1198.598551] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1202.296515] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1204.192881] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 1204.195816] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1204.200938] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1209.809826] Lustre: server umount lustre-MDT0001 complete [ 1210.338461] LustreError: 103991:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1210.344073] LustreError: 103991:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1212.900756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1212.932626] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1212.934387] Lustre: Found index 1 for lustre-MDT0001, updating log [ 1212.946962] Lustre: 103253:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1213.160316] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1213.163546] Lustre: Skipped 3 previous similar messages [ 1214.503562] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 1214.630128] LustreError: 107357:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 1214.632331] LustreError: 107357:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 1214.634770] LustreError: 103255:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 1214.670359] LustreError: 107363:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.109@tcp: cfg command failed: rc = -17 [ 1214.675091] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1214.675091] [ 1217.613728] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 01:08:56 (1782623336) [ 1218.537310] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1223.014539] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 1229.042229] Lustre: 109185:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1237.132942] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1238.351958] Lustre: 110316:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1285.088673] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1285.089271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1285.093668] Lustre: Skipped 4 previous similar messages [ 1285.095647] Lustre: Skipped 5 previous similar messages [ 1290.521350] Lustre: server umount lustre-MDT0000 complete [ 1291.643519] LustreError: 103245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782623410 with bad export cookie 2755979026729673416 [ 1291.644793] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1291.646550] LustreError: 103245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1291.754638] Lustre: server umount lustre-MDT0001 complete [ 1303.561343] Lustre: server umount lustre-OST0000 complete [ 1314.074493] Lustre: server umount lustre-OST0001 complete [ 1315.910576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1319.064739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1321.979556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1325.325389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1331.484396] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 1334.811876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1334.818612] Lustre: lustre-MDT0000: reset Object Index mappings [ 1334.950124] LustreError: 114241:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1334.958149] LustreError: 114241:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1334.972740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1336.169332] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 1338.692216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1338.698105] Lustre: lustre-MDT0001: reset Object Index mappings [ 1340.207867] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 1341.122255] Lustre: 115387:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1343.098862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1345.116687] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 1345.378449] LustreError: 115741:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1345.383661] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 1347.722228] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1348.907179] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 1348.907250] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 1348.934797] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 1349.728397] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 1352.726568] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1353.841201] Lustre: 117262:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1367.198171] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 01:11:26 (1782623486) [ 1400.291317] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1400.294539] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1400.298615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1400.299734] Lustre: Skipped 4 previous similar messages [ 1400.303279] Lustre: Skipped 9 previous similar messages [ 1402.983251] Lustre: server umount lustre-MDT0000 complete [ 1404.208293] LustreError: 115390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782623523 with bad export cookie 2755979026729720204 [ 1404.212265] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1404.214162] LustreError: 115390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1405.408939] LustreError: 114238:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1405.414680] LustreError: 114238:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1410.567189] Lustre: server umount lustre-MDT0001 complete [ 1426.400126] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1426.440896] Lustre: server umount lustre-OST0000 complete [ 1433.753597] Lustre: server umount lustre-OST0001 complete [ 1435.671807] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [ 1438.822867] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 1450.427885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1450.512351] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1450.526349] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1450.561945] Lustre: lustre-MDT0000: new disk, initializing [ 1450.591324] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1452.034616] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 1454.988874] Lustre: server umount lustre-MDT0000 complete [ 1458.011495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1458.101491] Lustre: Found index 0 for lustre-MDT0000, updating log [ 1458.106031] Lustre: 121521:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1458.111447] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 1459.401828] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 1461.000143] Lustre: server umount lustre-MDT0000 complete [ 1463.752919] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 01:13:03 (1782623583) [ 1464.272504] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 1464.816373] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 01:13:04 (1782623584) [ 1470.205281] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 1473.329473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1473.456460] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1473.459725] Lustre: Skipped 5 previous similar messages [ 1474.765530] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 1477.490772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1477.529949] Lustre: 123567:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1477.552904] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1477.554863] Lustre: Skipped 1 previous similar message [ 1477.592299] Lustre: lustre-MDT0001: new disk, initializing [ 1477.632127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1477.635795] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1478.930746] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 1481.091797] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 1483.052211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1483.145250] Lustre: lustre-OST0000: new disk, initializing [ 1483.146690] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1483.148693] Lustre: 125140:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1484.835814] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 1484.920960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1484.924631] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1484.953623] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1488.976743] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1489.046203] Lustre: 126012:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1490.241959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1490.245111] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1490.281207] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1490.990275] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 1495.255961] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1496.329588] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1609.906956] Lustre: 132384:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 1609.909682] Lustre: 132384:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 1609.912626] Lustre: 132384:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 1609.915728] Lustre: 132384:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 1609.918552] Lustre: 132384:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 1609.921537] Lustre: 132384:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1688.153905] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 01:16:47 (1782623807) [ 1702.880847] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1702.883884] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1702.888518] Lustre: Skipped 2 previous similar messages [ 1702.890707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1702.892564] Lustre: Skipped 4 previous similar messages [ 1705.841939] Lustre: server umount lustre-MDT0000 complete [ 1706.947061] LustreError: 132416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782623826 with bad export cookie 2755979026729772620 [ 1706.952046] LustreError: 132416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1706.952741] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1708.513129] LustreError: 126810:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1708.519952] LustreError: 126810:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1721.312159] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1721.422365] Lustre: server umount lustre-MDT0001 complete [ 1728.701244] Lustre: server umount lustre-OST0000 complete [ 1744.352139] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1744.420861] Lustre: server umount lustre-OST0001 complete [ 1746.392729] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [ 1749.245611] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 1760.559170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1760.635076] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1760.645644] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1760.647078] Lustre: Skipped 1 previous similar message [ 1760.679825] Lustre: lustre-MDT0000: new disk, initializing [ 1760.681089] Lustre: Skipped 1 previous similar message [ 1760.700080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1760.702131] Lustre: Skipped 3 previous similar messages [ 1760.707340] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1762.071211] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 1766.026950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1766.063526] Lustre: 135398:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1766.152425] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1767.362463] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 1770.930339] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1772.656874] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1774.975431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1775.112063] Lustre: 137368:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1776.901634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1776.904523] Lustre: Skipped 1 previous similar message [ 1776.906678] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1776.955679] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1777.307264] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 1781.199504] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1784.448446] Lustre: lustre-MDD0000: changelog on [ 1785.360428] Lustre: lustre-MDD0001: changelog on [ 2105.251067] Lustre: 135406:0:(llog_cat.c:977:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 2133.435472] Lustre: lustre-MDD0001: changelog off [ 2134.504613] Lustre: lustre-MDD0000: changelog off [ 2140.641812] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2140.642760] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2140.646027] LustreError: Skipped 1 previous similar message [ 2140.646346] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2140.646350] Lustre: Skipped 12 previous similar messages [ 2140.649930] Lustre: Skipped 6 previous similar messages [ 2150.368145] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2150.419392] Lustre: server umount lustre-OST0000 complete [ 2155.488857] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2157.760121] Lustre: server umount lustre-MDT0000 complete [ 2158.963331] LustreError: 137367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782624278 with bad export cookie 2755979026730577718 [ 2158.966540] LustreError: 137367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2158.967276] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2159.086254] Lustre: server umount lustre-MDT0001 complete [ 2162.486070] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 2163.444778] Key type lgssc unregistered [ 2163.559292] LNet: 146048:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2163.561336] LNetError: 146048:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2163.569234] LNet: Removed LNI 192.168.204.109@tcp [ 2163.855155] Key type .llcrypt unregistered [ 2163.856461] Key type ._llcrypt unregistered [ 2170.114253] Key type ._llcrypt registered [ 2170.115307] Key type .llcrypt registered [ 2170.149769] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [ 2176.230834] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 2176.573610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2176.630709] alg: No test for adler32 (adler32-zlib) [ 2177.478888] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 2177.568286] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 2179.144155] Key type lgssc registered [ 2179.470224] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2189.873234] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2189.878294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2190.962161] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2190.972646] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2191.012296] Lustre: lustre-MDT0000: new disk, initializing [ 2191.038226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2191.046089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2192.380435] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2196.340903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2196.383471] Lustre: 149223:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2196.409800] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2196.411372] Lustre: Skipped 1 previous similar message [ 2196.442204] Lustre: lustre-MDT0001: new disk, initializing [ 2196.474584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2196.487680] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2196.492181] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2197.766620] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2201.399332] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2203.023144] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2205.154517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2205.252324] Lustre: lustre-OST0000: new disk, initializing [ 2205.254279] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2205.256878] Lustre: 151187:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2205.282272] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2205.364660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2205.368405] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2205.403887] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2207.131152] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2210.692701] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2212.862799] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_module obdecho/obdecho [ 2213.477036] LustreError: 152190:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 2213.480205] LustreError: 152190:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 2214.056972] LustreError: 152286:0:(echo_client.c:1883:echo_md_handler()) Can not resolve the path : rc = -2 [ 2221.026705] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2221.027546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2221.030517] Lustre: Skipped 1 previous similar message [ 2223.388491] Lustre: server umount lustre-MDT0000 complete [ 2224.573134] LustreError: 149214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782624343 with bad export cookie 10356471034191984743 [ 2224.575710] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2224.579429] LustreError: 149214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2224.686510] Lustre: server umount lustre-MDT0001 complete [ 2236.193507] Lustre: server umount lustre-OST0000 complete [ 2243.212796] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [ 2246.516763] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 2257.799654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2257.878996] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2257.889060] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2257.922405] Lustre: lustre-MDT0000: new disk, initializing [ 2257.941273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2257.946917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2259.220095] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2263.221583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2263.261378] Lustre: 155991:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2263.283414] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2263.285096] Lustre: Skipped 1 previous similar message [ 2263.322564] Lustre: lustre-MDT0001: new disk, initializing [ 2263.352165] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2263.362474] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2263.366119] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2264.642434] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2268.157641] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2269.824340] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2271.944236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2272.042453] Lustre: lustre-OST0000: new disk, initializing [ 2272.044851] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2272.047596] Lustre: 157961:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2272.073079] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2273.467143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2273.471496] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2273.514680] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2273.909154] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2277.551231] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2278.868746] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2278.940483] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2280.271642] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2280.348604] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2284.001445] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2284.003488] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2284.004827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2284.006473] Lustre: Skipped 2 previous similar messages [ 2284.007946] Lustre: Skipped 2 previous similar messages [ 2287.010743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2287.012334] Lustre: Skipped 2 previous similar messages [ 2292.194346] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2292.196244] Lustre: Skipped 1 previous similar message [ 2295.264116] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2295.318493] Lustre: server umount lustre-OST0000 complete [ 2297.313630] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2297.314282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2297.318212] Lustre: Skipped 1 previous similar message [ 2302.609722] Lustre: server umount lustre-MDT0000 complete [ 2303.725113] LustreError: 155984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782624422 with bad export cookie 10356471034191985688 [ 2303.727512] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2303.728396] LustreError: 155984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2303.837971] Lustre: server umount lustre-MDT0001 complete [ 2304.903436] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 01:27:04 (1782624424) [ 2305.363608] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 2305.867445] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 01:27:05 (1782624425) [ 2308.149661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2308.314745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2309.527289] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2312.081266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2313.378469] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2315.468547] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2317.133490] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2319.123501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2321.085414] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2323.476923] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2335.810164] LustreError: 162968:0:(ldlm_resource.c:1207:ldlm_resource_complain()) mdt-lustre-MDT0000_UUID: namespace resource [0x200000bd1:0x2f:0x0].0x0 (000000000dcbab14) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2337.761066] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2337.764022] LustreError: Skipped 1 previous similar message [ 2337.765731] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2337.770829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2337.773050] Lustre: Skipped 3 previous similar messages [ 2341.975996] Lustre: server umount lustre-MDT0000 complete [ 2343.120025] LustreError: 160351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782624462 with bad export cookie 10356471034191986556 [ 2343.123371] LustreError: 160351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2343.123547] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2343.904769] LustreError: 160395:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2343.905043] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2343.908725] LustreError: 160395:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2343.917287] Lustre: Skipped 2 previous similar messages [ 2349.024785] LustreError: 160395:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2349.032816] LustreError: 160395:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2349.305556] Lustre: server umount lustre-MDT0001 complete [ 2351.606560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2355.913739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2355.995460] LustreError: 164229:0:(llog.c:1646:llog_backup()) MGC192.168.204.109@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 2355.998386] Lustre: 164229:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.109@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 2373.600908] LustreError: 164233:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2373.608092] LustreError: 164233:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2378.721034] LustreError: 164251:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2378.728393] LustreError: 164251:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2383.840835] LustreError: 164234:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2383.847914] LustreError: 164234:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2393.056740] LustreError: 164254:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2393.061976] LustreError: 164254:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2409.654564] LustreError: 164251:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2409.661945] LustreError: 164251:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 2416.736280] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 2416.739200] Lustre: 164229:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.204.109@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 2443.745303] LustreError: 164254:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2443.749703] LustreError: 164254:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 21 previous similar messages [ 2457.576761] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2457.580083] Lustre: Skipped 2 previous similar messages [ 2458.976260] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2461.775507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2462.694238] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2463.712734] LustreError: 147449:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000000480281b x1869217053902848/t0(0) o250->MGC192.168.204.109@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2465.175288] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2467.638761] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2468.221209] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2468.838629] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2470.048117] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2471.634735] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 1 sec [ 2478.161137] Lustre: server umount lustre-MDT0000 complete [ 2479.074538] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2479.078300] Lustre: Skipped 2 previous similar messages [ 2479.079646] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2479.228949] LustreError: 160352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782624598 with bad export cookie 10356471034191992996 [ 2479.232064] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2479.232277] LustreError: 160352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2479.327418] Lustre: server umount lustre-MDT0001 complete [ 2490.715401] Lustre: server umount lustre-OST0000 complete [ 2497.510993] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [ 2500.655031] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 2512.665696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2512.750819] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2512.763235] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2512.798702] Lustre: lustre-MDT0000: new disk, initializing [ 2512.823816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2514.232336] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2518.432312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2518.482277] Lustre: 169299:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2518.591329] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2520.010396] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2523.769479] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2525.551468] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2527.776791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2527.900509] Lustre: 171268:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2527.928080] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2527.930690] Lustre: Skipped 3 previous similar messages [ 2529.837692] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2533.503647] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2534.395876] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2534.442221] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2535.029746] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2535.119254] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2536.467571] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2536.540991] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2539.490903] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2539.491640] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2539.495492] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2539.500647] Lustre: Skipped 1 previous similar message [ 2539.503041] Lustre: Skipped 8 previous similar messages [ 2543.249240] Lustre: server umount lustre-OST0000 complete [ 2549.216981] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2549.219806] LustreError: Skipped 1 previous similar message [ 2550.628697] Lustre: server umount lustre-MDT0000 complete [ 2551.887404] LustreError: 169293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782624671 with bad export cookie 10356471034192009796 [ 2551.890900] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2551.892639] LustreError: 169293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2552.005830] Lustre: server umount lustre-MDT0001 complete [ 2554.884238] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 01:31:14 (1782624674) [ 2557.070052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2557.211137] LustreError: 173677:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2557.217178] LustreError: 173677:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 20 previous similar messages [ 2558.613428] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2561.483209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2562.928649] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2565.304888] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2567.194484] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2569.428309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2571.417253] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2573.748600] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2588.239358] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 01:31:47 (1782624707) [ 2592.227121] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2592.227717] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2592.231366] LustreError: Skipped 1 previous similar message [ 2592.235489] Lustre: Skipped 2 previous similar messages [ 2592.236060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2592.240339] Lustre: Skipped 5 previous similar messages [ 2595.211500] Lustre: server umount lustre-OST0000 complete [ 2602.584603] Lustre: server umount lustre-MDT0000 complete [ 2603.918581] LustreError: 173657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782624723 with bad export cookie 10356471034192010664 [ 2603.924349] LustreError: 173657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2603.924932] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2604.050901] Lustre: server umount lustre-MDT0001 complete [ 2607.905240] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 2608.992384] Key type lgssc unregistered [ 2609.132350] LNet: 177423:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2609.135074] LNetError: 177423:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2609.142366] LNet: Removed LNI 192.168.204.109@tcp [ 2609.439389] Key type .llcrypt unregistered [ 2609.440387] Key type ._llcrypt unregistered [ 2616.585596] Key type ._llcrypt registered [ 2616.587020] Key type .llcrypt registered [ 2616.625151] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 2616.993793] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2617.002706] alg: No test for adler32 (adler32-zlib) [ 2617.847427] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 2617.932045] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 2619.512150] Key type lgssc registered [ 2619.884343] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2622.416148] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2622.422712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2637.856393] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2642.976259] LustreError: 178644:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.109@tcp: failed processing log, type 4: rc = -110 [ 2648.096345] LustreError: 178644:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 2648.099901] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 2648.101723] LustreError: MGC192.168.204.109@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 2648.104987] LustreError: 178644:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 2648.107552] LustreError: 178644:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 2648.109825] LustreError: 178644:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2648.126757] Lustre: server umount lustre-OST0000 complete [ 2648.128041] LustreError: 178644:0:(super25.c:185:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 2650.818151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2651.948811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2653.243649] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2656.021809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2656.146876] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2657.508095] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2659.783811] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2661.537296] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2663.609678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2663.679619] LustreError: 180957:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 2663.684298] Lustre: 180957:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.109@tcp: invalid local config log lustre-OST0000: rc = -22 [ 2663.731130] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2665.638397] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2668.032631] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2671.584891] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2671.588897] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2671.594581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2672.097444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2672.099245] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2677.218788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2677.221662] Lustre: Skipped 4 previous similar messages [ 2682.338592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2682.340448] Lustre: Skipped 2 previous similar messages [ 2683.360108] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2683.417090] Lustre: server umount lustre-MDT0000 complete [ 2684.644300] LustreError: 179122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782624803 with bad export cookie 13571565705497540776 [ 2684.647896] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2684.649441] LustreError: 179122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2684.760953] Lustre: server umount lustre-MDT0001 complete [ 2696.342766] Lustre: server umount lustre-OST0000 complete [ 2699.359362] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [ 2702.684338] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 2714.553395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2714.637747] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2714.648135] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2714.685725] Lustre: lustre-MDT0000: new disk, initializing [ 2714.707422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2714.713631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2716.152496] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2720.355236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2720.395937] Lustre: 184472:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2720.419601] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2720.421908] Lustre: Skipped 1 previous similar message [ 2720.461256] Lustre: lustre-MDT0001: new disk, initializing [ 2720.495740] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2720.509797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2720.514607] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2721.891256] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2725.611959] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2727.487167] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2729.741640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2729.844578] Lustre: lustre-OST0000: new disk, initializing [ 2729.846430] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2729.849135] Lustre: 186438:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2729.876620] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2731.813096] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2735.515164] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2736.120554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2736.125322] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2736.155399] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2737.089107] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2737.157772] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2738.425703] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2738.491223] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2741.217394] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2741.217616] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2741.220713] LustreError: Skipped 1 previous similar message [ 2741.225038] Lustre: Skipped 1 previous similar message [ 2741.225520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2741.229131] Lustre: Skipped 1 previous similar message [ 2745.176996] Lustre: server umount lustre-OST0000 complete [ 2746.337322] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2746.337775] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2746.341963] Lustre: Skipped 2 previous similar messages [ 2751.456942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2751.458766] Lustre: Skipped 3 previous similar messages [ 2752.473897] Lustre: server umount lustre-MDT0000 complete [ 2753.606162] LustreError: 184466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782624872 with bad export cookie 13571565705497541476 [ 2753.609719] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2753.610626] LustreError: 184466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2753.712675] Lustre: server umount lustre-MDT0001 complete [ 2756.261692] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 01:34:35 (1782624875) [ 2761.662888] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 2762.620966] Key type lgssc unregistered [ 2762.734319] LNet: 189382:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2762.736502] LNetError: 189382:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2762.745249] LNet: Removed LNI 192.168.204.109@tcp [ 2763.029526] Key type .llcrypt unregistered [ 2763.031073] Key type ._llcrypt unregistered [ 2770.002546] Key type ._llcrypt registered [ 2770.004008] Key type .llcrypt registered [ 2770.041127] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 2770.417316] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2770.432748] alg: No test for adler32 (adler32-zlib) [ 2771.272799] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 2771.358748] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 2772.936153] Key type lgssc registered [ 2773.243974] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2775.554282] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2775.560724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2776.688896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2777.973061] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2780.573352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2780.678994] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2781.989462] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2784.145158] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2785.811691] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2788.157241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2788.233221] Lustre: *** cfs_fail_loc=131e, val=0*** [ 2788.234970] LustreError: 192441:0:(llog.c:1687:llog_backup()) MGC192.168.204.109@tcp: failed to backup log lustre-OST0000: rc = -28 [ 2788.238672] Lustre: 192441:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.109@tcp: can't backup local config lustre-OST0000: rc = -28 [ 2788.273889] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2790.098837] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2792.312638] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2792.955996] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2792.957203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2792.960032] Lustre: Skipped 2 previous similar messages [ 2792.961995] Lustre: Skipped 1 previous similar message [ 2798.049456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2798.051101] Lustre: Skipped 2 previous similar messages [ 2803.169847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2803.172592] Lustre: Skipped 2 previous similar messages [ 2807.264088] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2807.313207] Lustre: server umount lustre-MDT0000 complete [ 2808.288944] LustreError: 192030:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2808.292657] LustreError: 192030:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2808.384878] LustreError: 190559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782624927 with bad export cookie 12431121241154059076 [ 2808.387678] LustreError: 190559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2808.387728] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2808.487409] Lustre: server umount lustre-MDT0001 complete [ 2818.904866] Lustre: server umount lustre-OST0000 complete [ 2822.058665] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [ 2824.891819] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 2835.769985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2835.848613] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2835.858304] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2835.890480] Lustre: lustre-MDT0000: new disk, initializing [ 2835.908861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2835.914207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2837.164415] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2841.056207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2841.095037] Lustre: 195957:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2841.113463] Lustre: *** cfs_fail_loc=131e, val=0*** [ 2841.115114] Lustre: Skipped 1 previous similar message [ 2841.116852] LustreError: 196740:0:(llog.c:1687:llog_backup()) MGC192.168.204.109@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 2841.120479] LustreError: 196740:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 2841.123237] Lustre: 196740:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.109@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 2841.131753] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2841.133496] Lustre: Skipped 1 previous similar message [ 2841.164928] Lustre: lustre-MDT0001: new disk, initializing [ 2841.191960] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2841.203291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2841.206780] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2842.430038] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2845.889947] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2847.557038] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2849.695026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2849.785941] Lustre: *** cfs_fail_loc=131e, val=0*** [ 2849.787634] Lustre: Skipped 1 previous similar message [ 2849.789274] LustreError: 197901:0:(llog.c:1687:llog_backup()) MGC192.168.204.109@tcp: failed to backup log lustre-OST0000: rc = -28 [ 2849.791668] LustreError: 197901:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 2849.794048] Lustre: 197901:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.109@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 2849.796897] Lustre: 197901:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 2849.804927] Lustre: lustre-OST0000: new disk, initializing [ 2849.807020] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2849.810992] Lustre: 197924:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2849.840444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2850.208796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2850.211625] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2850.245474] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2851.653541] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2855.181666] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2856.549702] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2856.617493] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2857.975723] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2858.045962] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2860.513514] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2860.514281] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2860.515943] LustreError: Skipped 1 previous similar message [ 2860.522263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2860.524615] Lustre: Skipped 1 previous similar message [ 2860.686552] Lustre: server umount lustre-OST0000 complete [ 2862.048872] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2862.052056] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2862.056633] Lustre: Skipped 1 previous similar message [ 2865.633407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2865.634064] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2865.636152] Lustre: Skipped 1 previous similar message [ 2875.873198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2875.876405] Lustre: Skipped 3 previous similar messages [ 2876.384094] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2876.443123] Lustre: server umount lustre-MDT0000 complete [ 2877.749323] LustreError: 195950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782624996 with bad export cookie 12431121241154059762 [ 2877.751827] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2877.754899] LustreError: 195950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2877.874442] Lustre: server umount lustre-MDT0001 complete [ 2880.680630] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 01:36:40 (1782625000) [ 2886.037888] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 2889.659734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2889.825037] LustreError: 201044:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2889.838156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2891.234137] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2893.988269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2895.500270] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2896.469353] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2898.718584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2900.891589] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2903.813526] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2903.890529] Lustre: lustre-OST0001: new disk, initializing [ 2903.893031] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2903.896851] Lustre: 203349:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2905.934114] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2906.116426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2906.121398] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2906.156339] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2911.372424] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2917.111067] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2917.751592] Lustre: DEBUG MARKER: ADD OST3 [ 2919.541835] loop: module loaded [ 2919.562040] loop0: detected capacity change from 0 to 204800000 [ 2919.647167] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 2919.652649] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 2919.658623] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 2921.898087] loop0: detected capacity change from 0 to 204800000 [ 2923.885232] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2923.954194] Lustre: lustre-OST0003: new disk, initializing [ 2923.956813] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 2923.960762] Lustre: 205081:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2924.001808] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 2924.004638] Lustre: Skipped 3 previous similar messages [ 2924.686259] Lustre: DEBUG MARKER: STOP OST3 [ 2925.426270] Lustre: Failing over lustre-OST0003 [ 2925.458984] Lustre: server umount lustre-OST0003 complete [ 2925.570394] LustreError: 202710:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2926.357728] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2930.165115] Lustre: DEBUG MARKER: START OST3 again [ 2930.874444] LustreError: 202527:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2930.879244] LustreError: 202527:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2932.633737] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2934.789435] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2937.847992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 2937.852356] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 2937.864471] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000401 [ 2948.327557] Lustre: Failing over lustre-OST0003 [ 2948.373291] Lustre: server umount lustre-OST0003 complete [ 2953.186939] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2953.189439] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2953.193709] LustreError: 202530:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2953.194000] Lustre: Skipped 1 previous similar message [ 2953.200112] LustreError: 202530:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2960.908309] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2960.910747] LustreError: Skipped 1 previous similar message [ 2960.912616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2963.427066] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2963.429886] LustreError: 202530:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2963.432349] Lustre: Skipped 3 previous similar messages [ 2963.443631] LustreError: 202530:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2975.200091] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2975.271955] Lustre: server umount lustre-MDT0000 complete [ 2976.464222] LustreError: 202178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782625095 with bad export cookie 12431121241154060630 [ 2976.467509] LustreError: 202178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2976.468073] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2981.792766] LustreError: 207157:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2981.796982] LustreError: 207157:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2982.706752] Lustre: server umount lustre-MDT0001 complete [ 2984.106220] Lustre: server umount lustre-OST0000 complete [ 2985.437094] Lustre: server umount lustre-OST0001 complete [ 2987.572247] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [ 2990.269541] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 3002.230394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3002.310823] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3002.321681] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3002.365166] Lustre: lustre-MDT0000: new disk, initializing [ 3002.387836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3002.390937] Lustre: Skipped 1 previous similar message [ 3002.396746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3003.752266] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3007.820275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3007.866318] Lustre: 210502:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3007.971269] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3009.329735] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3013.029685] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3014.876892] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3017.166408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3017.273634] Lustre: 212469:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3019.134974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3019.138385] Lustre: Skipped 1 previous similar message [ 3019.140800] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3019.187272] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3019.388484] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3023.149455] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3024.611350] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3024.697132] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3026.104917] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3026.187945] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3029.474978] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3029.475597] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3029.478413] LustreError: Skipped 1 previous similar message [ 3029.482931] Lustre: Skipped 2 previous similar messages [ 3029.483481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3029.489095] Lustre: Skipped 17 previous similar messages [ 3041.248065] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3041.297844] Lustre: server umount lustre-OST0000 complete [ 3043.808905] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3048.603493] Lustre: server umount lustre-MDT0000 complete [ 3049.868575] LustreError: 211296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782625169 with bad export cookie 12431121241154062863 [ 3049.873306] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3049.874135] LustreError: 211296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3049.997426] Lustre: server umount lustre-MDT0001 complete [ 3051.170721] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 01:39:30 (1782625170) [ 3056.148541] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [ 3059.306625] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 3070.695530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3070.787446] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3070.798844] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3070.800649] Lustre: Skipped 3 previous similar messages [ 3070.835117] Lustre: lustre-MDT0000: new disk, initializing [ 3070.836537] Lustre: Skipped 2 previous similar messages [ 3070.860400] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3072.152506] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3076.203552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3076.243993] Lustre: 217070:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3076.343706] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3077.658611] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3081.336178] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3083.072513] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3085.278149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3085.377667] Lustre: 219038:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3087.398995] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3087.400819] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3091.030774] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3092.510493] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3092.595269] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3094.028096] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3094.111383] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3097.570656] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3097.571270] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3097.577652] Lustre: Skipped 3 previous similar messages [ 3097.582435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3097.585308] Lustre: Skipped 8 previous similar messages [ 3109.344128] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3109.397878] Lustre: server umount lustre-OST0000 complete [ 3116.765607] Lustre: server umount lustre-MDT0000 complete [ 3117.974108] LustreError: 217064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782625237 with bad export cookie 12431121241154063745 [ 3117.977609] LustreError: 217064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3117.977865] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3118.095104] Lustre: server umount lustre-MDT0001 complete [ 3120.864586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3121.013278] LustreError: 221255:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3122.369486] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3125.219179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3126.707636] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3129.037092] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3130.779316] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3132.994664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3133.111696] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3133.114529] Lustre: Skipped 7 previous similar messages [ 3134.999386] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3137.290418] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3155.937758] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3155.940166] LustreError: Skipped 2 previous similar messages [ 3160.143340] Lustre: server umount lustre-OST0000 complete [ 3163.106431] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3163.112767] Lustre: Skipped 4 previous similar messages [ 3167.513575] Lustre: server umount lustre-MDT0000 complete [ 3168.717274] LustreError: 221235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782625287 with bad export cookie 12431121241154064627 [ 3168.719614] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3168.723054] LustreError: 221235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3168.831813] Lustre: server umount lustre-MDT0001 complete [ 3270.848278] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 3271.860522] Key type lgssc unregistered [ 3272.001491] LNet: 224935:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3272.004673] LNetError: 224935:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3272.014379] LNet: Removed LNI 192.168.204.109@tcp [ 3272.375091] Key type .llcrypt unregistered [ 3272.376413] Key type ._llcrypt unregistered [ 3280.625796] Key type ._llcrypt registered [ 3280.628485] Key type .llcrypt registered [ 3280.664646] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [ 3287.015305] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 3287.293456] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3287.305718] alg: No test for adler32 (adler32-zlib) [ 3288.159190] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 3288.249941] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 3289.832127] Key type lgssc registered [ 3290.198141] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3301.532496] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3301.541364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3302.621725] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3302.631476] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3302.671125] Lustre: lustre-MDT0000: new disk, initializing [ 3302.707363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3302.714626] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3303.993180] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3307.963016] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3308.003261] Lustre: 228262:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3308.026572] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3308.028348] Lustre: Skipped 1 previous similar message [ 3308.063547] Lustre: lustre-MDT0001: new disk, initializing [ 3308.093793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3308.105276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3308.108937] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3309.386571] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3312.937968] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3314.619349] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3316.729935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3316.831213] Lustre: lustre-OST0000: new disk, initializing [ 3316.833406] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3316.836722] Lustre: 230226:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3316.863905] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3318.265949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3318.269441] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3318.307217] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3318.708721] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3322.304387] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3323.739389] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3323.815428] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3325.148575] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3325.220814] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3328.481918] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3328.484853] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3328.486994] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3328.489619] Lustre: Skipped 1 previous similar message [ 3333.603899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3333.606511] Lustre: Skipped 2 previous similar messages [ 3338.723549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3338.725686] Lustre: Skipped 1 previous similar message [ 3340.256122] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3340.306554] Lustre: server umount lustre-OST0000 complete [ 3343.841713] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3343.841890] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3343.842784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3343.845561] Lustre: Skipped 1 previous similar message [ 3343.849180] LustreError: Skipped 1 previous similar message [ 3347.675922] Lustre: server umount lustre-MDT0000 complete [ 3348.935540] LustreError: 228253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782625468 with bad export cookie 10637120508156462538 [ 3348.939026] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3348.941649] LustreError: 228253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3349.061792] Lustre: server umount lustre-MDT0001 complete [ 3355.161122] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 3358.557147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3358.696747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3360.016230] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3362.852459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3362.976341] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3364.333342] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3365.362735] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3367.499449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3369.447666] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3372.196045] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3372.276790] Lustre: lustre-OST0001: new disk, initializing [ 3372.279068] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3372.281533] Lustre: 235463:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3372.320744] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3372.323425] Lustre: Skipped 1 previous similar message [ 3374.287229] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3379.205728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3379.209667] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3379.246203] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3379.763988] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3384.825627] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3388.732769] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 01:45:08 (1782625508) [ 3389.420959] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3389.424548] Lustre: Skipped 1 previous similar message [ 3389.425965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3389.428050] Lustre: Skipped 1 previous similar message [ 3394.017525] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3394.020182] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3394.023760] Lustre: Skipped 1 previous similar message [ 3395.618106] Lustre: server umount lustre-MDT0000 complete [ 3396.792131] LustreError: 234292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782625515 with bad export cookie 10637120508156463406 [ 3396.795589] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3396.797058] LustreError: 234292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3396.911465] Lustre: server umount lustre-MDT0001 complete [ 3408.897984] Lustre: server umount lustre-OST0000 complete [ 3420.376655] Lustre: server umount lustre-OST0001 complete [ 3422.380984] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [ 3425.279742] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 3436.490082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3436.579498] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3436.589711] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3436.632784] Lustre: lustre-MDT0000: new disk, initializing [ 3436.653450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3436.659384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3437.968396] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3442.046040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3442.090961] Lustre: 239404:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3442.197298] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3443.482536] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3447.069490] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3448.823815] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3451.132838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3451.229830] Lustre: lustre-OST0000: new disk, initializing [ 3451.231200] Lustre: Skipped 1 previous similar message [ 3451.232689] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3451.234352] Lustre: Skipped 2 previous similar messages [ 3451.236323] Lustre: 241370:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3452.545753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3452.549763] Lustre: Skipped 1 previous similar message [ 3452.552427] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3452.592201] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3453.245644] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3457.043992] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3458.418922] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3458.487148] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3459.878922] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3459.950745] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3463.141466] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3463.141924] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3463.143189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3463.143206] Lustre: Skipped 6 previous similar messages [ 3463.144178] LustreError: Skipped 1 previous similar message [ 3463.154642] Lustre: Skipped 1 previous similar message [ 3466.639362] Lustre: server umount lustre-OST0000 complete [ 3474.035308] Lustre: server umount lustre-MDT0000 complete [ 3475.427087] LustreError: 241369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782625594 with bad export cookie 10637120508156465527 [ 3475.429879] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3475.430582] LustreError: 241369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3475.539161] Lustre: server umount lustre-MDT0001 complete [ 3478.229405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3478.380760] LustreError: 243583:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3478.386430] LustreError: 243583:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3478.404743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3478.406736] Lustre: Skipped 2 previous similar messages [ 3479.703656] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3482.477297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3483.925862] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3486.174254] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3487.885335] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3490.146591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3492.112668] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3494.438906] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3640.800845] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3640.803249] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3640.806929] Lustre: Skipped 2 previous similar messages [ 3640.809014] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3640.810773] Lustre: Skipped 5 previous similar messages [ 3651.040077] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3651.090316] Lustre: server umount lustre-OST0000 complete [ 3656.672842] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3656.675825] LustreError: Skipped 1 previous similar message [ 3657.185980] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3657.189414] Lustre: Skipped 2 previous similar messages [ 3658.394323] Lustre: server umount lustre-MDT0000 complete [ 3659.565256] LustreError: 243563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782625778 with bad export cookie 10637120508156466409 [ 3659.568210] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3659.570651] LustreError: 243563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3659.694625] Lustre: server umount lustre-MDT0001 complete [ 3758.817115] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 3759.899240] Key type lgssc unregistered [ 3760.048322] LNet: 247194:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3760.051128] LNetError: 247194:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3760.058312] LNet: Removed LNI 192.168.204.109@tcp [ 3760.412096] Key type .llcrypt unregistered [ 3760.413063] Key type ._llcrypt unregistered [ 3768.389100] Key type ._llcrypt registered [ 3768.390021] Key type .llcrypt registered [ 3768.426544] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [ 3774.482837] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 3774.881124] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3774.894914] alg: No test for adler32 (adler32-zlib) [ 3775.739687] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 3775.820233] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 3777.400140] Key type lgssc registered [ 3777.775983] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3791.972817] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 3795.095754] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3795.102958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3796.175479] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3796.186532] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3796.219108] Lustre: lustre-MDT0000: new disk, initializing [ 3796.239978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3796.245052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3797.498546] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3801.489250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3801.525440] Lustre: 251234:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3801.545991] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3801.547868] Lustre: Skipped 1 previous similar message [ 3801.579455] Lustre: lustre-MDT0001: new disk, initializing [ 3801.607692] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3801.620636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3801.625453] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3802.849546] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3805.055487] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3807.225978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3807.316162] Lustre: lustre-OST0000: new disk, initializing [ 3807.318702] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3807.321916] Lustre: 252864:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3807.354361] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3809.215822] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3810.422602] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3810.426129] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3810.462286] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3813.193205] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3813.252993] Lustre: lustre-OST0001: new disk, initializing [ 3813.255298] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3813.257592] Lustre: 253738:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3813.291823] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3815.078683] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3817.980821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3817.983608] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3818.013662] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3820.375268] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3825.987524] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3832.288809] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3832.291184] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3832.295080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3833.314232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3833.314958] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3833.321134] Lustre: Skipped 1 previous similar message [ 3834.916478] Lustre: server umount lustre-MDT0000 complete [ 3836.095050] LustreError: 251226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782625955 with bad export cookie 5560220750635293762 [ 3836.097665] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3836.098509] LustreError: 251226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3836.218713] Lustre: server umount lustre-MDT0001 complete [ 3847.620108] Lustre: server umount lustre-OST0000 complete [ 3858.077342] Lustre: server umount lustre-OST0001 complete [ 3862.703190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3865.787638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3868.383438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3871.382757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3877.780860] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 3881.060088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3881.067040] Lustre: lustre-MDT0000: reset Object Index mappings [ 3881.208375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3882.512610] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3885.240345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3885.246554] Lustre: lustre-MDT0001: reset Object Index mappings [ 3886.715314] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3887.635317] Lustre: 259877:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3889.702909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3889.823626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3889.825739] Lustre: Skipped 1 previous similar message [ 3890.851145] LustreError: 260228:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3890.854612] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3890.856720] LustreError: 260228:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3891.635266] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3894.240636] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3895.337894] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 3896.127948] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3899.015798] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3900.136246] Lustre: 261742:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3905.508640] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3905.510810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3905.512494] Lustre: Skipped 4 previous similar messages [ 3905.514147] Lustre: Skipped 5 previous similar messages [ 3907.361242] Lustre: server umount lustre-MDT0000 complete [ 3908.508075] LustreError: 258716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782626027 with bad export cookie 5560220750635300538 [ 3908.510821] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3908.511787] LustreError: 258716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3908.615710] Lustre: server umount lustre-MDT0001 complete [ 3920.848216] Lustre: server umount lustre-OST0000 complete [ 3932.384170] Lustre: server umount lustre-OST0001 complete [ 3940.286795] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 01:54:19 (1782626059) [ 3947.545297] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [ 3950.948812] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 3965.180559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3965.257576] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3965.266222] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3965.299827] Lustre: lustre-MDT0000: new disk, initializing [ 3965.317874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3965.320120] Lustre: Skipped 1 previous similar message [ 3965.324472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3966.597021] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3971.423805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3971.461050] Lustre: 266000:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3971.555239] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3972.743595] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3976.915144] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3979.279424] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3981.983949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3982.128521] Lustre: lustre-OST0000: new disk, initializing [ 3982.130708] Lustre: Skipped 1 previous similar message [ 3982.133114] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3982.135534] Lustre: Skipped 2 previous similar messages [ 3982.138469] Lustre: 267968:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3983.372289] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3983.377941] Lustre: Skipped 1 previous similar message [ 3983.381723] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3983.432759] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3984.660311] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3989.250748] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3990.652827] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3990.737359] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3992.137201] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3992.211708] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3993.570941] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3993.572073] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3993.573251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3993.573532] LustreError: Skipped 1 previous similar message [ 3993.576570] Lustre: Skipped 1 previous similar message [ 3998.691169] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3998.693093] Lustre: Skipped 2 previous similar messages [ 4007.392175] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4007.478716] Lustre: server umount lustre-OST0000 complete [ 4008.930424] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4008.931710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4008.934963] Lustre: Skipped 1 previous similar message [ 4008.936895] Lustre: Skipped 2 previous similar messages [ 4023.264187] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4023.368518] Lustre: server umount lustre-MDT0000 complete [ 4024.289040] LustreError: 266801:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4024.293251] LustreError: 266801:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4024.675187] LustreError: 265993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782626143 with bad export cookie 5560220750635302610 [ 4024.678068] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4024.678375] LustreError: 265993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4024.798027] Lustre: server umount lustre-MDT0001 complete [ 4030.944626] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 4035.191227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4035.338330] LustreError: 270897:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4035.353938] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4035.357502] Lustre: Skipped 2 previous similar messages [ 4036.753668] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 4039.395529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4041.169066] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 4042.342583] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4045.293278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4047.867293] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 4051.297230] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4051.392433] Lustre: lustre-OST0001: new disk, initializing [ 4051.394290] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4051.396684] Lustre: 273202:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4053.195891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4053.198701] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4053.239618] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4053.626282] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 4058.924725] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4060.415602] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4062.303634] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 4062.315733] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 4065.250275] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4065.255717] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4065.265090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4065.269037] Lustre: Skipped 5 previous similar messages [ 4070.273515] Lustre: server umount lustre-MDT0000 complete [ 4072.004883] LustreError: 270924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782626191 with bad export cookie 5560220750635303492 [ 4072.010076] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4072.014795] LustreError: 270924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4072.174695] Lustre: server umount lustre-MDT0001 complete [ 4084.226912] Lustre: server umount lustre-OST0000 complete [ 4095.648861] Lustre: server umount lustre-OST0001 complete [ 4100.498916] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 4103.526218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4103.651720] LustreError: 276253:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4103.666330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4103.668256] Lustre: Skipped 3 previous similar messages [ 4104.904068] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 4107.554616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4108.950264] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 4109.862279] Lustre: 277393:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4111.846318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4112.996994] LustreError: 277747:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4113.001882] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 4113.706666] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 4116.303808] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4117.421342] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 4118.183355] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 4121.060487] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4122.117502] Lustre: 279267:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4123.644205] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 4123.645965] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 4126.246885] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 01:57:25 (1782626245) [ 4137.955201] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4137.957528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4137.961244] Lustre: Skipped 4 previous similar messages [ 4137.963831] Lustre: Skipped 4 previous similar messages [ 4143.711733] Lustre: server umount lustre-MDT0000 complete [ 4144.806303] LustreError: 276236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782626263 with bad export cookie 5560220750635309771 [ 4144.810032] LustreError: 276236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4144.810078] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4148.192932] LustreError: 276250:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4148.198033] LustreError: 276250:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4150.959298] Lustre: server umount lustre-MDT0001 complete [ 4158.203840] Lustre: server umount lustre-OST0000 complete [ 4165.584562] Lustre: server umount lustre-OST0001 complete [ 4167.714561] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [ 4170.521640] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 4184.095055] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 4187.202569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4187.282402] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4187.293557] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4187.327089] Lustre: lustre-MDT0000: new disk, initializing [ 4187.350584] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4188.607285] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 4192.683093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4192.720053] Lustre: 283499:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4192.824253] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4194.112450] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 4198.202074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4198.297749] Lustre: 285132:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4199.389939] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4200.042414] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 4203.817242] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4205.603376] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4205.645212] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 4209.822877] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4214.374484] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 01:58:53 (1782626333) [ 4215.779754] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4215.779754] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4215.779769] Lustre: Skipped 4 previous similar messages [ 4215.781046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4215.781052] Lustre: Skipped 11 previous similar messages [ 4215.786709] Lustre: Skipped 5 previous similar messages [ 4221.145281] Lustre: server umount lustre-MDT0000 complete [ 4222.222199] LustreError: 284292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782626341 with bad export cookie 5560220750635317653 [ 4222.225311] LustreError: 284292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4222.225690] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4222.327996] Lustre: server umount lustre-MDT0001 complete [ 4233.218403] Lustre: server umount lustre-OST0000 complete [ 4244.633310] Lustre: server umount lustre-OST0001 complete [ 4247.346349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4247.493478] LustreError: 288240:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4247.502765] LustreError: 288240:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4247.522632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4247.525798] Lustre: Skipped 7 previous similar messages [ 4248.766333] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 4251.346486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4252.742613] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 4254.839085] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4256.392911] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4258.507806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4260.340067] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 4262.457406] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4280.386816] loop1: detected capacity change from 0 to 204800000 [ 4280.476230] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4280.479549] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4280.483177] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4282.344227] LustreError: 291114:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4282.354089] LustreError: 291114:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 4282.725966] loop1: detected capacity change from 0 to 204800000 [ 4284.975449] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4285.008705] LustreError: 288231:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) fs15350: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 4285.013801] LustreError: 288231:0:(mgs_llog.c:731:mgs_check_index()) Can't get db for fs15350 [ 4285.015605] LustreError: 288231:0:(mgs_handler.c:517:mgs_target_reg()) fs15350-MDT0001: Error checking target index: rc = -13 [ 4285.018505] LustreError: fs15350-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 4285.021156] LustreError: 291976:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 4285.023545] LustreError: 291976:0:(tgt_mount.c:2129:server_put_super()) no obd fs15350-MDT0001 [ 4285.025872] LustreError: 291976:0:(tgt_mount.c:116:server_deregister_mount()) fs15350-MDT0001 not registered [ 4285.038064] Lustre: server umount fs15350-MDT0001 complete [ 4285.040512] LustreError: 291976:0:(super25.c:185:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 4288.265330] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4288.294580] Lustre: Setting parameter fs15350-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15350-MDT0001 [ 4288.298398] Lustre: Skipped 2 previous similar messages [ 4288.314027] Lustre: srv-fs15350-MDT0001: No data found on store. Initialize space. [ 4288.317372] Lustre: Skipped 4 previous similar messages [ 4288.360562] Lustre: fs15350-MDT0001: new disk, initializing [ 4288.362861] Lustre: Skipped 3 previous similar messages [ 4288.402977] Lustre: srv-fs15350-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 4288.408516] Lustre: 292442:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs15350-MDT0001-mdtlov: can't create new llog: rc = -115 [ 4288.414954] LustreError: 292442:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs15350-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 4290.027654] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 4293.517807] Lustre: server umount fs15350-MDT0001 complete [ 4299.542653] loop1: detected capacity change from 0 to 204800000 [ 4302.127135] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4302.293655] Lustre: 294006:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs15350-MDT0001-mdtlov: can't create new llog: rc = -115 [ 4302.297565] LustreError: 294006:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs15350-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 4303.861742] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 4306.160858] Lustre: server umount fs15350-MDT0001 complete [ 4313.060118] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4313.060409] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4313.068294] Lustre: Skipped 2 previous similar messages [ 4316.343900] Lustre: server umount lustre-MDT0000 complete [ 4320.051542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4320.112512] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4321.797497] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 4325.358417] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4325.435592] loop1: detected capacity change from 0 to 204800000 [ 4327.914343] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4328.074677] Lustre: 296449:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs15350-MDT0001-mdtlov: can't create new llog: rc = -115 [ 4328.078246] LustreError: 296449:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs15350-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 4329.850150] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 4332.206684] Lustre: server umount fs15350-MDT0001 complete [ 4343.777517] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4343.783576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4343.787533] Lustre: Skipped 11 previous similar messages [ 4347.370761] Lustre: server umount lustre-OST0000 complete [ 4355.383252] Lustre: server umount lustre-MDT0000 complete [ 4356.066970] LustreError: 288970:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4356.076514] LustreError: 288970:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 41 previous similar messages [ 4357.222962] LustreError: 296940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782626476 with bad export cookie 5560220750635320327 [ 4357.228344] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4357.230794] LustreError: 296940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4357.397568] Lustre: server umount lustre-MDT0001 complete [ 4362.345769] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 4363.501809] Key type lgssc unregistered [ 4363.617683] LNet: 298800:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4363.621797] LNetError: 298800:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4363.633489] LNet: Removed LNI 192.168.204.109@tcp [ 4364.032149] Key type .llcrypt unregistered [ 4364.033807] Key type ._llcrypt unregistered [ 4373.601365] Key type ._llcrypt registered [ 4373.603186] Key type .llcrypt registered [ 4373.661546] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 4374.138861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4374.155762] alg: No test for adler32 (adler32-zlib) [ 4375.015086] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 4375.108876] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 4376.688222] Key type lgssc registered [ 4377.271169] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4380.602970] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4380.614689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4381.793470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4383.205478] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 4386.968396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4387.150697] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4388.945432] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 4392.228291] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4394.446092] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4397.746168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4397.952660] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4400.564161] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 4404.003523] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4406.183738] LustreError: 301874:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4406.198895] LustreError: 301874:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4410.539938] LustreError: 301874:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4412.391053] LustreError: 301875:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4412.398534] LustreError: 301875:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4415.674572] LustreError: 301876:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4420.792450] LustreError: 301874:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4420.799133] LustreError: 301874:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4422.050541] loop1: detected capacity change from 0 to 204800000 [ 4422.184875] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4422.192591] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4422.200631] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4424.683910] loop1: detected capacity change from 0 to 204800000 [ 4427.178450] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4427.252953] Lustre: fs2187-OST0000: new disk, initializing [ 4427.255525] Lustre: srv-fs2187-OST0000: No data found on store. Initialize space. [ 4427.259896] Lustre: 303802:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4427.293054] Lustre: fs2187-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4429.502244] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 4431.031733] LustreError: 302548:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4431.036233] LustreError: 302548:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4432.859930] Lustre: server umount fs2187-OST0000 complete [ 4436.461532] loop1: detected capacity change from 0 to 204800000 [ 4438.153207] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4438.213174] Lustre: fs2187-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4439.924053] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 4443.566492] loop2: detected capacity change from 0 to 204800000 [ 4443.640412] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4443.643731] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4443.647165] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4445.416114] loop2: detected capacity change from 0 to 204800000 [ 4447.985491] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4448.037386] LustreError: fs2187-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 4448.042982] LustreError: fs2187-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 4448.049687] LustreError: 306837:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 4448.055273] LustreError: 306837:0:(tgt_mount.c:2129:server_put_super()) no obd fs2187-OST0001 [ 4448.060126] LustreError: 306837:0:(tgt_mount.c:116:server_deregister_mount()) fs2187-OST0001 not registered [ 4448.078774] Lustre: server umount fs2187-OST0001 complete [ 4448.081614] LustreError: 306837:0:(super25.c:185:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 4448.228354] LustreError: 301875:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4448.239536] LustreError: 301875:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 4452.989548] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4453.074390] Lustre: fs2187-OST0001: new disk, initializing [ 4453.079485] Lustre: srv-fs2187-OST0001: No data found on store. Initialize space. [ 4453.086740] Lustre: 307387:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4453.131137] Lustre: fs2187-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4455.582114] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 4460.139203] Lustre: server umount fs2187-OST0000 complete [ 4464.731269] Lustre: server umount fs2187-OST0001 complete [ 4470.921664] loop1: detected capacity change from 0 to 204800000 [ 4473.198772] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4473.267834] Lustre: fs2187-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4475.689210] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 4478.763775] loop2: detected capacity change from 0 to 204800000 [ 4481.345850] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4482.234091] LustreError: 301876:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4482.245156] LustreError: 301876:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 18 previous similar messages [ 4483.956785] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 4487.814421] Lustre: server umount fs2187-OST0000 complete [ 4489.190374] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [ 4491.520991] Lustre: server umount fs2187-OST0001 complete [ 4498.400791] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4498.403068] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4498.406678] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4499.426436] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4499.427562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4499.428588] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4504.545135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4504.547973] Lustre: Skipped 2 previous similar messages [ 4509.152174] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4509.216659] Lustre: server umount lustre-OST0000 complete [ 4514.785567] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4514.786080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4514.788953] Lustre: Skipped 1 previous similar message [ 4516.825057] Lustre: server umount lustre-MDT0000 complete [ 4517.969065] LustreError: 301899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782626637 with bad export cookie 8374943164986622925 [ 4517.972596] LustreError: 301899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4517.973184] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4518.081530] Lustre: server umount lustre-MDT0001 complete [ 4521.347620] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 4522.245985] Key type lgssc unregistered [ 4522.361268] LNet: 313299:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4522.363561] LNetError: 313299:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4522.372277] LNet: Removed LNI 192.168.204.109@tcp [ 4522.642089] Key type .llcrypt unregistered [ 4522.643139] Key type ._llcrypt unregistered [ 4531.880090] Key type ._llcrypt registered [ 4531.881175] Key type .llcrypt registered [ 4531.913503] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [ 4538.570348] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 4538.900521] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4538.943801] alg: No test for adler32 (adler32-zlib) [ 4539.813895] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 4539.926090] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 4541.536214] Key type lgssc registered [ 4542.116081] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4556.685098] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4556.692634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4557.783115] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4557.794255] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4557.840473] Lustre: lustre-MDT0000: new disk, initializing [ 4557.860060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4557.866857] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4559.249287] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 4563.720870] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4563.758852] Lustre: 317130:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4563.781440] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4563.783009] Lustre: Skipped 1 previous similar message [ 4563.817223] Lustre: lustre-MDT0001: new disk, initializing [ 4563.845873] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4563.855737] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4563.858758] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4565.494168] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 4570.089365] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4572.522640] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4575.600605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4575.730362] Lustre: lustre-OST0000: new disk, initializing [ 4575.733751] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4575.738271] Lustre: 319098:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4575.780099] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4576.653677] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4576.659569] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4576.704988] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4578.155955] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 4582.188757] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4582.917930] Lustre: Mounted lustre-client [ 4583.243092] Lustre: Unmounted lustre-client [ 4586.977859] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4586.977996] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4586.983280] LustreError: Skipped 1 previous similar message [ 4586.990966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4586.997364] Lustre: Skipped 1 previous similar message [ 4588.004442] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4588.009166] Lustre: Skipped 1 previous similar message [ 4591.013246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4591.016665] Lustre: Skipped 1 previous similar message [ 4592.661552] Lustre: server umount lustre-OST0000 complete [ 4594.657294] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4594.660928] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4594.666199] Lustre: Skipped 1 previous similar message [ 4594.668590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4596.194982] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4599.900418] Lustre: server umount lustre-MDT0000 complete [ 4601.196125] LustreError: 317121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782626720 with bad export cookie 1361573522155179439 [ 4601.198346] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4601.201397] LustreError: 317121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4601.303558] Lustre: server umount lustre-MDT0001 complete [ 4605.761921] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 4606.978106] Key type lgssc unregistered [ 4644.960343] LNet: 321258:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4644.962684] LNetError: 321258:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4711.526534] LNet: Removed LNI 192.168.204.109@tcp [ 4711.978798] Key type .llcrypt unregistered [ 4711.980215] Key type ._llcrypt unregistered [ 4719.797153] Key type ._llcrypt registered [ 4719.798923] Key type .llcrypt registered [ 4719.856522] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [ 4727.566962] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 4727.958788] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4728.002159] alg: No test for adler32 (adler32-zlib) [ 4728.852876] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 4728.942857] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 4730.528183] Key type lgssc registered [ 4731.182254] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4745.129450] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 02:07:44 (1782626864) [ 4747.809269] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4747.820933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4748.941298] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4748.954543] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4749.012102] Lustre: lustre-MDT0000: new disk, initializing [ 4749.043791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4749.055240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4750.849604] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 4755.637073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4755.702221] Lustre: 324455:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4755.735389] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4755.738081] Lustre: Skipped 1 previous similar message [ 4755.781529] Lustre: lustre-MDT0001: new disk, initializing [ 4755.819987] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4755.833696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4755.837947] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4757.540395] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 4762.332103] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4765.120991] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4768.444813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4768.605186] Lustre: lustre-OST0000: new disk, initializing [ 4768.608797] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4768.613899] Lustre: 326422:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4768.666314] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4771.391145] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 4774.390083] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4774.393098] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4774.426047] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4775.448386] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4784.616744] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4784.617742] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4784.620733] LustreError: Skipped 1 previous similar message [ 4784.628277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4784.633765] Lustre: Skipped 1 previous similar message [ 4788.598591] Lustre: server umount lustre-OST0000 complete [ 4791.777501] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4791.782598] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4791.790565] Lustre: Skipped 1 previous similar message [ 4791.794172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4794.851123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4794.852201] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4796.619993] Lustre: server umount lustre-MDT0000 complete [ 4798.498410] LustreError: 324448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782626917 with bad export cookie 4027246638395918176 [ 4798.506072] LustreError: 324448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4798.508234] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4798.664302] Lustre: server umount lustre-MDT0001 complete [ 4803.939478] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 4805.537280] Key type lgssc unregistered [ 4805.734575] LNet: 328329:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4805.739747] LNetError: 328329:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4805.748749] LNet: Removed LNI 192.168.204.109@tcp [ 4806.236875] Key type .llcrypt unregistered [ 4806.238807] Key type ._llcrypt unregistered [ 4817.241736] Key type ._llcrypt registered [ 4817.243087] Key type .llcrypt registered [ 4817.290400] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 4817.601351] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4817.635045] alg: No test for adler32 (adler32-zlib) [ 4818.497291] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 4818.595753] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 4820.184167] Key type lgssc registered [ 4820.624754] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4824.235290] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4824.247515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4825.456354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4827.304660] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 4831.535088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4831.727694] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4833.551659] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 4836.915823] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4839.621785] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4842.891409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4843.026507] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4845.658300] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 4849.264846] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4863.993430] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4864.858125] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4867.553456] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4867.558535] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4867.567380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4871.363171] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.9@tcp (stopping) [ 4871.655367] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4876.472453] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.9@tcp (stopping) [ 4876.478223] Lustre: Skipped 3 previous similar messages [ 4881.376121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4881.464843] Lustre: server umount lustre-MDT0000 complete [ 4881.592174] LustreError: 332077:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4881.603084] LustreError: 332077:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4883.305189] LustreError: 331422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782627002 with bad export cookie 2213444991628882579 [ 4883.311919] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4883.312803] LustreError: 331422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4886.710751] LustreError: 332077:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4886.715759] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.9@tcp (stopping) [ 4886.716097] LustreError: 332077:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4886.717983] Lustre: Skipped 3 previous similar messages [ 4887.009914] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4887.014300] Lustre: Skipped 1 previous similar message [ 4888.033362] LustreError: 329575:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4888.043543] LustreError: 329575:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4889.529050] Lustre: server umount lustre-MDT0001 complete [ 4891.163465] Lustre: server umount lustre-OST0000 complete [ 4895.005824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4896.923072] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 4898.235515] Lustre: 333780:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.109@tcp [ 4899.260822] Lustre: server umount MGS complete [ 4903.348960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4903.558325] LustreError: 334351:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4903.591196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4905.382385] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 4909.026983] LustreError: 334352:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4909.037913] LustreError: 334352:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4909.535794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4909.693493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4911.423596] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 4915.534435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4918.298182] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 4921.645455] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4922.406926] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [ 4924.968914] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4933.116086] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 7 sec [ 4935.137350] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4935.142374] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4935.151174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4935.155392] Lustre: Skipped 2 previous similar messages [ 4938.214177] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4940.708117] Lustre: server umount lustre-OST0000 complete [ 4943.330857] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4943.332208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4943.339276] Lustre: Skipped 2 previous similar messages [ 4943.343310] Lustre: Skipped 2 previous similar messages [ 4948.658743] Lustre: server umount lustre-MDT0000 complete [ 4950.450235] LustreError: 335808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782627069 with bad export cookie 2213444991628883398 [ 4950.457172] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4950.457801] LustreError: 335808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4950.599200] Lustre: server umount lustre-MDT0001 complete [ 4955.061917] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 4956.538977] Key type lgssc unregistered [ 4956.696614] LNet: 337787:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4956.701820] LNetError: 337787:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4956.712583] LNet: Removed LNI 192.168.204.109@tcp [ 4957.188682] Key type .llcrypt unregistered [ 4957.190908] Key type ._llcrypt unregistered [ 4965.717748] Key type ._llcrypt registered [ 4965.720027] Key type .llcrypt registered [ 4965.792389] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [ 4974.236389] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 4974.744279] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4974.760277] alg: No test for adler32 (adler32-zlib) [ 4975.673404] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 4975.796802] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 4977.408176] Key type lgssc registered [ 4977.974840] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4995.274487] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4995.283607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4996.381859] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4996.393158] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4996.440191] Lustre: lustre-MDT0000: new disk, initializing [ 4996.472505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4996.483051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4998.437450] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 5004.157677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5004.223585] Lustre: 340919:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5004.267581] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5004.271496] Lustre: Skipped 1 previous similar message [ 5004.322336] Lustre: lustre-MDT0001: new disk, initializing [ 5004.371451] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5004.391788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5004.398985] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5006.417786] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 5011.264206] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5013.885933] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5016.714949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5016.883078] Lustre: lustre-OST0000: new disk, initializing [ 5016.886802] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5016.892262] Lustre: 342890:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5016.950929] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5019.842690] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 5023.254578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5023.261443] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5023.332401] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5024.658579] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5026.638932] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5026.756233] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5028.689501] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5028.794898] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5033.448000] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5033.450170] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5033.452647] LustreError: Skipped 1 previous similar message [ 5033.461489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5033.467291] Lustre: Skipped 1 previous similar message [ 5035.826394] Lustre: server umount lustre-OST0000 complete [ 5038.564729] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5038.566424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5038.572979] Lustre: Skipped 2 previous similar messages [ 5038.579592] Lustre: Skipped 1 previous similar message [ 5043.682384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5043.688165] Lustre: Skipped 1 previous similar message [ 5043.853818] Lustre: server umount lustre-MDT0000 complete [ 5045.726408] LustreError: 340914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782627164 with bad export cookie 4047210297893176392 [ 5045.733839] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5045.733994] LustreError: 340914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5045.926506] Lustre: server umount lustre-MDT0001 complete [ 5050.114211] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 02:12:49 (1782627169) [ 5057.674425] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [ 5061.846562] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 5078.255599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5078.386481] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5078.401775] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5078.452043] Lustre: lustre-MDT0000: new disk, initializing [ 5078.480801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5078.489849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5080.235875] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 5085.805095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5085.870980] Lustre: 347643:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5085.913938] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5085.918300] Lustre: Skipped 1 previous similar message [ 5085.965878] Lustre: lustre-MDT0001: new disk, initializing [ 5086.014616] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5086.031360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5086.036133] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5087.935833] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 5092.821661] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5095.413678] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5098.710334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5098.898762] Lustre: lustre-OST0000: new disk, initializing [ 5098.901117] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5098.904641] Lustre: 349609:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5098.955844] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5101.308861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5101.311855] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5101.352066] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5101.510288] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 5106.307423] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5122.022082] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5122.022774] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5122.027214] LustreError: Skipped 1 previous similar message [ 5122.033630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5122.038520] Lustre: Skipped 1 previous similar message [ 5124.204422] Lustre: server umount lustre-OST0000 complete [ 5127.139905] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5127.140131] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5127.150876] Lustre: Skipped 1 previous similar message [ 5127.154878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5127.157948] Lustre: Skipped 1 previous similar message [ 5132.097351] Lustre: server umount lustre-MDT0000 complete [ 5133.912718] LustreError: 347635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782627253 with bad export cookie 4047210297893177260 [ 5133.919620] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5133.920167] LustreError: 347635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5134.088574] Lustre: server umount lustre-MDT0001 complete [ 5138.762320] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 5140.167290] Key type lgssc unregistered [ 5140.321584] LNet: 351757:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5140.326873] LNetError: 351757:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5140.336643] LNet: Removed LNI 192.168.204.109@tcp [ 5140.793349] Key type .llcrypt unregistered [ 5140.795743] Key type ._llcrypt unregistered [ 5183.671736] Key type ._llcrypt registered [ 5183.672873] Key type .llcrypt registered [ 5183.708130] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 5183.813311] Key type .llcrypt unregistered [ 5183.814579] Key type ._llcrypt unregistered [ 5219.372991] Key type ._llcrypt registered [ 5219.374227] Key type .llcrypt registered [ 5219.421271] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 5219.552726] Key type .llcrypt unregistered [ 5219.555100] Key type ._llcrypt unregistered [ 5232.840709] Key type ._llcrypt registered [ 5232.842451] Key type .llcrypt registered [ 5232.890929] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 5233.053294] Key type .llcrypt unregistered [ 5233.055705] Key type ._llcrypt unregistered [ 5246.766868] Key type ._llcrypt registered [ 5246.767889] Key type .llcrypt registered [ 5246.828742] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 5246.979308] Key type .llcrypt unregistered [ 5246.981755] Key type ._llcrypt unregistered [ 5280.286361] Key type ._llcrypt registered [ 5280.288030] Key type .llcrypt registered [ 5280.353332] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 5280.492263] Key type .llcrypt unregistered [ 5280.494042] Key type ._llcrypt unregistered [ 5294.243764] Key type ._llcrypt registered [ 5294.245706] Key type .llcrypt registered [ 5294.320508] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 5294.512343] Key type .llcrypt unregistered [ 5294.515158] Key type ._llcrypt unregistered [ 5308.558740] Key type ._llcrypt registered [ 5308.560577] Key type .llcrypt registered [ 5308.623551] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 5308.780379] Key type .llcrypt unregistered [ 5308.782486] Key type ._llcrypt unregistered [ 5321.734306] Key type ._llcrypt registered [ 5321.735695] Key type .llcrypt registered [ 5321.790474] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 5321.967407] Key type .llcrypt unregistered [ 5321.969568] Key type ._llcrypt unregistered [ 5335.044105] Key type ._llcrypt registered [ 5335.045158] Key type .llcrypt registered [ 5335.085133] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 5335.181356] Key type .llcrypt unregistered [ 5335.182382] Key type ._llcrypt unregistered [ 5349.731681] Key type ._llcrypt registered [ 5349.733709] Key type .llcrypt registered [ 5349.794461] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 5349.951664] Key type .llcrypt unregistered [ 5349.953940] Key type ._llcrypt unregistered [ 5378.760364] Key type ._llcrypt registered [ 5378.762315] Key type .llcrypt registered [ 5378.837347] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 5379.020525] Key type .llcrypt unregistered [ 5379.023105] Key type ._llcrypt unregistered [ 5394.248971] Key type ._llcrypt registered [ 5394.251197] Key type .llcrypt registered [ 5394.318099] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 5394.495549] Key type .llcrypt unregistered [ 5394.497840] Key type ._llcrypt unregistered [ 5409.217622] Key type ._llcrypt registered [ 5409.219510] Key type .llcrypt registered [ 5409.271128] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 5409.418330] Key type .llcrypt unregistered [ 5409.420534] Key type ._llcrypt unregistered [ 5433.744087] Key type ._llcrypt registered [ 5433.745835] Key type .llcrypt registered [ 5433.789145] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [ 5442.619136] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 5443.002167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5443.019206] alg: No test for adler32 (adler32-zlib) [ 5443.883192] Lustre: Lustre: Build Version: 2.17.54_105_g3b83550 [ 5443.997238] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 5445.600149] Key type lgssc registered [ 5446.190491] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5462.029189] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5462.039700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5463.157363] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 5463.175559] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 5463.224553] Lustre: tst250fs-MDT0000: new disk, initializing [ 5463.248956] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5463.256833] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5465.043396] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 5470.507716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5470.580192] Lustre: 371701:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5470.622366] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 5470.626968] Lustre: Skipped 1 previous similar message [ 5470.675446] Lustre: tst250fs-MDT0001: new disk, initializing [ 5470.710259] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5470.722595] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5470.727713] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5472.419743] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 5475.251463] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5478.466464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5478.621145] Lustre: tst250fs-OST0000: new disk, initializing [ 5478.623619] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 5478.627348] Lustre: 373337:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5478.665437] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5481.257937] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 5485.585922] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5485.593690] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5485.660706] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5486.949666] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5487.034387] Lustre: tst250fs-OST0001: new disk, initializing [ 5487.037364] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 5487.041990] Lustre: 374205:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5487.088103] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5489.640948] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 5492.796705] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 5494.122544] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5494.125654] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5494.168793] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5506.529680] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5506.534736] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5506.543517] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 5509.607329] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 5509.607513] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5509.612634] Lustre: Skipped 2 previous similar messages [ 5509.620650] Lustre: Skipped 2 previous similar messages [ 5511.477816] Lustre: server umount tst250fs-MDT0000 complete [ 5513.197485] LustreError: 372498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782627632 with bad export cookie 8884164840440265581 [ 5513.204301] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5513.204974] LustreError: 372498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5513.353676] Lustre: server umount tst250fs-MDT0001 complete [ 5525.334032] Lustre: server umount tst250fs-OST0000 complete [ 5537.470492] Lustre: server umount tst250fs-OST0001 complete [ 5546.084795] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [ 5549.824561] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 5563.836331] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 02:21:22 (1782627682) [ 5564.702171] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 5572.468105] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [ 5576.954234] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 5591.876372] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 5459 sec ========== 02:21:50 (1782627710) [ 5592.837693] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 02:21:51 (1782627711) === [ 5594.551338] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 02:21:53 (1782627713) === [ 5612.867181] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 5614.221642] Key type lgssc unregistered [ 5614.381555] LNet: 383340:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5614.386598] LNetError: 383340:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5614.395590] LNet: Removed LNI 192.168.204.109@tcp [ 5614.871159] Key type .llcrypt unregistered [ 5614.873621] Key type ._llcrypt unregistered