[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 520899806 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002351] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.007325] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009010] pid_max: default: 32768 minimum: 301 [ 0.010180] LSM: Security Framework initializing [ 0.011053] Yama: becoming mindful. [ 0.012030] SELinux: Initializing. [ 0.012839] *** VALIDATE selinux *** [ 0.020806] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025423] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026156] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027093] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028108] *** VALIDATE tmpfs *** [ 0.029392] *** VALIDATE proc *** [ 0.031084] *** VALIDATE cgroup *** [ 0.031904] *** VALIDATE cgroup2 *** [ 0.033057] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034125] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036025] Spectre V2 : User space: Vulnerable [ 0.037007] Speculative Store Bypass: Vulnerable [ 0.040095] debug: unmapping init [mem 0xffffffffa9a59000-0xffffffffa9a60fff] [ 0.042752] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043687] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044025] ... version: 2 [ 0.045012] ... bit width: 48 [ 0.046013] ... generic registers: 4 [ 0.047013] ... value mask: 0000ffffffffffff [ 0.048022] ... max period: 00007fffffffffff [ 0.049021] ... fixed-purpose events: 3 [ 0.050015] ... event mask: 000000070000000f [ 0.051288] rcu: Hierarchical SRCU implementation. [ 0.053615] smp: Bringing up secondary CPUs ... [ 0.054520] x86: Booting SMP configuration: [ 0.055020] .... node #0, CPUs: #1 #2 #3 [ 0.058070] smp: Brought up 1 node, 4 CPUs [ 0.060010] smpboot: Max logical packages: 1 [ 0.061014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.109017] node 0 deferred pages initialised in 46ms [ 0.112331] devtmpfs: initialized [ 0.113249] x86/mm: Memory block size: 128MB [ 0.115933] gcov: version magic: 0x41383552 [ 0.116640] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.120090] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.122377] pinctrl core: initialized pinctrl subsystem [ 0.124144] [ 0.124739] ************************************************************* [ 0.127011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.129007] ** ** [ 0.131009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.132008] ** ** [ 0.134009] ** This means that this kernel is built to expose internal ** [ 0.136008] ** IOMMU data structures, which may compromise security on ** [ 0.138013] ** your system. ** [ 0.140012] ** ** [ 0.142010] ** If you see this message and you are not debugging the ** [ 0.145011] ** kernel, report this immediately to your vendor! ** [ 0.147010] ** ** [ 0.150015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.153015] ************************************************************* [ 0.156786] NET: Registered protocol family 16 [ 0.158528] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.161059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.164066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.168077] cpuidle: using governor menu [ 0.169926] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.172881] PCI: Using configuration type 1 for base access [ 0.175133] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.186154] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.188025] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.193104] cryptd: max_cpu_qlen set to 1000 [ 0.195257] ACPI: Added _OSI(Module Device) [ 0.196047] ACPI: Added _OSI(Processor Device) [ 0.197016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.199014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.204919] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.212450] ACPI: Interpreter enabled [ 0.214046] ACPI: PM: (supports S0 S3 S4 S5) [ 0.216014] ACPI: Using IOAPIC for interrupt routing [ 0.218093] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.221625] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.231819] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.234050] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.237020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.240087] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.246489] acpiphp: Slot [2] registered [ 0.248107] acpiphp: Slot [5] registered [ 0.249099] acpiphp: Slot [6] registered [ 0.250078] acpiphp: Slot [7] registered [ 0.252105] acpiphp: Slot [8] registered [ 0.253075] acpiphp: Slot [9] registered [ 0.255109] acpiphp: Slot [10] registered [ 0.256095] acpiphp: Slot [3] registered [ 0.258096] acpiphp: Slot [4] registered [ 0.259095] acpiphp: Slot [11] registered [ 0.261184] acpiphp: Slot [12] registered [ 0.263099] acpiphp: Slot [13] registered [ 0.264098] acpiphp: Slot [14] registered [ 0.266094] acpiphp: Slot [15] registered [ 0.267088] acpiphp: Slot [16] registered [ 0.269087] acpiphp: Slot [17] registered [ 0.270109] acpiphp: Slot [18] registered [ 0.272084] acpiphp: Slot [19] registered [ 0.273100] acpiphp: Slot [20] registered [ 0.275103] acpiphp: Slot [21] registered [ 0.277146] acpiphp: Slot [22] registered [ 0.278092] acpiphp: Slot [23] registered [ 0.280112] acpiphp: Slot [24] registered [ 0.282101] acpiphp: Slot [25] registered [ 0.283117] acpiphp: Slot [26] registered [ 0.285130] acpiphp: Slot [27] registered [ 0.287096] acpiphp: Slot [28] registered [ 0.288111] acpiphp: Slot [29] registered [ 0.290091] acpiphp: Slot [30] registered [ 0.292085] acpiphp: Slot [31] registered [ 0.293060] PCI host bridge to bus 0000:00 [ 0.295017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.297022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.300023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.303021] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.306027] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.309052] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.311173] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.314091] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.318334] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.328790] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.333524] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.336019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.339025] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.342018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.345606] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.348972] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.352033] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.354877] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.360015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.373015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.378014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.383335] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.390015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.398018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.422020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.433194] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.442015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.449018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.466027] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.479686] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.487011] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.493014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.508024] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.518110] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.525013] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.535018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.554028] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.566246] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.572017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.582014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.599019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.611222] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.634029] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.643024] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.669022] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.681429] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.684428] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.686436] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.690516] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.693376] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.698110] iommu: Default domain type: Passthrough [ 0.701330] SCSI subsystem initialized [ 0.702209] ACPI: bus type USB registered [ 0.704174] usbcore: registered new interface driver usbfs [ 0.706153] usbcore: registered new interface driver hub [ 0.709106] usbcore: registered new device driver usb [ 0.711222] pps_core: LinuxPPS API ver. 1 registered [ 0.713016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.716088] PTP clock support registered [ 0.720117] EDAC MC: Ver: 3.0.0 [ 0.722126] PCI: Using ACPI for IRQ routing [ 0.724251] NetLabel: Initializing [ 0.726016] NetLabel: domain hash size = 128 [ 0.728013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.730149] NetLabel: unlabeled traffic allowed by default [ 0.733134] vgaarb: loaded [ 0.735356] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.737017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.743853] clocksource: Switched to clocksource kvm-clock [ 0.861751] VFS: Disk quotas dquot_6.6.0 [ 0.864134] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.867215] *** VALIDATE ramfs *** [ 0.868554] *** VALIDATE hugetlbfs *** [ 0.870351] pnp: PnP ACPI init [ 0.873139] pnp: PnP ACPI: found 6 devices [ 0.895226] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.899027] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.901326] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.903797] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.906278] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.909097] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.912429] NET: Registered protocol family 2 [ 0.915138] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.920489] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.924555] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.934175] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.938287] TCP: Hash tables configured (established 65536 bind 65536) [ 0.941644] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.945300] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.948403] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.951846] NET: Registered protocol family 1 [ 0.954733] RPC: Registered named UNIX socket transport module. [ 0.958068] RPC: Registered udp transport module. [ 0.960236] RPC: Registered tcp transport module. [ 0.962372] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.964443] NET: Registered protocol family 44 [ 0.966345] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.968792] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.970738] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.973266] PCI: CLS 0 bytes, default 64 [ 0.975338] Unpacking initramfs... [ 2.422403] debug: unmapping init [mem 0xffff893afcc54000-0xffff893afffbffff] [ 2.427060] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.429639] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.433214] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.959781] Initialise system trusted keyrings [ 2.961914] Key type blacklist registered [ 2.964226] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.975172] zbud: loaded [ 2.978556] *** VALIDATE nfs *** [ 2.980355] *** VALIDATE nfs4 *** [ 2.982415] pstore: using deflate compression [ 2.986194] Platform Keyring initialized [ 3.116960] NET: Registered protocol family 38 [ 3.118825] Key type asymmetric registered [ 3.120230] Asymmetric key parser 'x509' registered [ 3.121914] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.124829] io scheduler mq-deadline registered [ 3.126697] io scheduler kyber registered [ 3.128370] io scheduler bfq registered [ 3.130540] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.133692] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.136750] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.139940] ACPI: Power Button [PWRF] [ 3.146749] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.153780] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.172972] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.184097] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.206000] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.233436] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.261976] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.268978] Non-volatile memory driver v1.3 [ 3.271176] Linux agpgart interface v0.103 [ 3.308354] virtio_blk virtio1: [vda] 146664 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.310959] vda: detected capacity change from 0 to 75091968 [ 3.335099] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.338975] vdb: detected capacity change from 0 to 1073741824 [ 3.369106] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.372677] vdc: detected capacity change from 0 to 2621440000 [ 3.387608] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.391088] vdd: detected capacity change from 0 to 2621440000 [ 3.411257] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.414378] vde: detected capacity change from 0 to 4294967296 [ 3.433874] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.437020] vdf: detected capacity change from 0 to 4294967296 [ 3.451567] libphy: Fixed MDIO Bus: probed [ 3.465482] usbcore: registered new interface driver usbserial_generic [ 3.467896] usbserial: USB Serial support registered for generic [ 3.470327] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.475508] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.477866] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.480773] mousedev: PS/2 mouse device common for all mice [ 3.484040] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.490212] rtc_cmos 00:05: RTC can wake from S4 [ 3.493903] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.498670] rtc_cmos 00:05: registered as rtc0 [ 3.499792] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.500730] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.507728] intel_pstate: CPU model not supported [ 3.510950] hid: raw HID events driver (C) Jiri Kosina [ 3.513646] usbcore: registered new interface driver usbhid [ 3.515884] usbhid: USB HID core driver [ 3.517645] drop_monitor: Initializing network drop monitor service [ 3.520563] Initializing XFRM netlink socket [ 3.522750] NET: Registered protocol family 10 [ 3.526209] Segment Routing with IPv6 [ 3.527972] NET: Registered protocol family 17 [ 3.530537] mpls_gso: MPLS GSO support [ 3.535688] RAS: Correctable Errors collector initialized. [ 3.537908] AVX version of gcm_enc/dec engaged. [ 3.539476] AES CTR mode by8 optimization enabled [ 3.619628] sched_clock: Marking stable (3619609410, 0)->(4531740236, -912130826) [ 3.623528] registered taskstats version 1 [ 3.625835] Loading compiled-in X.509 certificates [ 3.628393] zswap: loaded using pool lzo/zbud [ 3.658859] Key type big_key registered [ 3.673261] Key type encrypted registered [ 3.675426] ima: No TPM chip found, activating TPM-bypass! [ 3.678407] ima: Allocated hash algorithm: sha1 [ 3.680816] ima: No architecture policies found [ 3.682809] evm: Initialising EVM extended attributes: [ 3.685092] evm: security.selinux [ 3.686462] evm: security.ima [ 3.687738] evm: security.capability [ 3.689377] evm: HMAC attrs: 0x1 [ 3.692383] rtc_cmos 00:05: setting system clock to 2026-09-13 01:46:04 UTC (1789263964) [ 3.698756] debug: unmapping init [mem 0xffffffffaaa03000-0xffffffffaabfffff] [ 3.702332] debug: unmapping init [mem 0xffffffffa9782000-0xffffffffa9a58fff] [ 3.712067] Write protecting the kernel read-only data: 28672k [ 3.716039] debug: unmapping init [mem 0xffffffffa7e03000-0xffffffffa7ffffff] [ 3.720017] debug: unmapping init [mem 0xffffffffa8714000-0xffffffffa87fffff] [ 3.765780] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.776567] systemd[1]: Detected virtualization kvm. [ 3.778771] systemd[1]: Detected architecture x86-64. [ 3.781115] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.809751] systemd[1]: No hostname configured. [ 3.811889] systemd[1]: Set hostname to . [ 3.814301] random: systemd: uninitialized urandom read (16 bytes read) [ 3.817399] systemd[1]: Initializing machine ID from random generator. [ 3.851680] random: ln: uninitialized urandom read (6 bytes read) [ 3.947492] random: systemd: uninitialized urandom read (16 bytes read) [ 3.949838] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.955671] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.965619] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Reached target Local Encrypted Volumes. Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.606392] device-mapper: uevent: version 1.0.3 [ 4.609186] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.442303] virtio_net virtio0 ens2: renamed from eth0 [ 5.467184] random: fast init done [ 5.534273] scsi host0: ata_piix [ 5.551888] scsi host1: ata_piix [ 5.553595] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.556184] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.891453] dracut-initqueue[596]: RTNETLINK answers: File exists [ 10.197372] random: crng init done [ 10.198757] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.707810] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.938903] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.156024] SELinux: Disabled at runtime. [ 12.208756] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.218242] systemd[1]: Detected virtualization kvm. [ 12.220322] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.688298] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.691897] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.696689] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.701681] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.704931] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.712242] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.719491] systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Huge Pages File System... Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Initrd Root File System. Starting Remount Root and Kernel File Systems... Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-getty.slice. [ 12.900460] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.195812] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.459544] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.511655] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.586406] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.606235] EDAC sbridge: Ver: 1.1.2 [ 15.009880] Key type dns_resolver registered [ 15.327315] NFS: Registering the id_resolver key type [ 15.331232] Key type id_resolver registered [ 15.332914] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg314-server login: [ 41.192667] libcfs: loading out-of-tree module taints kernel. [ 41.212766] Key type ._llcrypt registered [ 41.214583] Key type .llcrypt registered [ 41.258772] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 50.563590] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 51.589492] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 51.602676] alg: No test for adler32 (adler32-zlib) [ 52.832636] Lustre: Lustre: Build Version: 2.17.57_104_g543d5f9 [ 53.478237] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 55.151796] Key type lgssc registered [ 56.307636] Lustre: Echo OBD driver; http://www.lustre.org/ [ 67.651747] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 91.292847] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 99.247174] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 99.276034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 100.454452] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 100.479697] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 100.531302] Lustre: lustre-MDT0000: new disk, initializing [ 100.578861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 100.593242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 103.395661] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 112.227413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 112.297770] Lustre: 6492:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 112.327524] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 112.331479] Lustre: Skipped 1 previous similar message [ 112.387767] Lustre: lustre-MDT0001: new disk, initializing [ 112.435223] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 112.458550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 112.465705] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 114.780146] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 118.205472] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 123.362266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 123.500466] Lustre: lustre-OST0000: new disk, initializing [ 123.503870] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 123.507917] Lustre: 8430:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 123.543060] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 124.556988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 124.563422] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 124.596355] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 127.024762] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 135.802066] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 135.877449] Lustre: lustre-OST0001: new disk, initializing [ 135.879627] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 135.883212] Lustre: 9502:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 135.912790] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 139.014210] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 143.889933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 143.898171] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 143.927807] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 147.297876] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 152.319344] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 157.150357] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing check_logdir /tmp/testlogs/ [ 160.695209] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing yml_node [ 163.706671] Lustre: DEBUG MARKER: Client: 2.17.57.104 [ 165.521660] Lustre: DEBUG MARKER: MDS: 2.17.57.104 [ 167.352501] Lustre: DEBUG MARKER: OSS: 2.17.57.104 [ 168.746972] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Sep 12 21:48:47 EDT 2026 [ 181.662746] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 182.817744] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 199.648801] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 199.652883] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 199.667298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 200.161615] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 200.168886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 200.177265] Lustre: Skipped 2 previous similar messages [ 200.185858] Lustre: Skipped 3 previous similar messages [ 204.933706] Lustre: server umount lustre-MDT0000 complete [ 207.504794] LustreError: 8440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789264168 with bad export cookie 14641862161375978872 [ 207.506764] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 207.513825] LustreError: 8440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 210.400450] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 210.400635] LustreError: 6500:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 210.409966] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 210.427711] LustreError: 6500:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 210.432566] Lustre: Skipped 1 previous similar message [ 212.447873] LustreError: 9978:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 212.447997] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 212.460386] LustreError: 9978:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 212.474852] Lustre: Skipped 1 previous similar message [ 213.791663] Lustre: server umount lustre-MDT0001 complete [ 216.281642] Lustre: server umount lustre-OST0000 complete [ 218.847372] Lustre: server umount lustre-OST0001 complete [ 223.778404] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 230.962544] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 258.215392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 258.447160] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 258.476911] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 258.613402] Lustre: lustre-MDT0000: new disk, initializing [ 258.708052] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 258.728097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 262.542459] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 271.553369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 271.674065] Lustre: 15044:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 271.757120] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 271.764137] Lustre: Skipped 1 previous similar message [ 271.834721] Lustre: lustre-MDT0001: new disk, initializing [ 271.944157] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 271.984884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 271.999578] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 275.573174] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 283.471958] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 288.240507] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 294.003107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 294.275571] Lustre: lustre-OST0000: new disk, initializing [ 294.283483] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 294.295352] Lustre: 17011:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 294.382146] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 295.610923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 295.620560] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 295.738064] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 299.954569] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 306.848711] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 310.470616] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 310.679738] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 313.763408] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 313.918342] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 316.385104] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 316.390908] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 316.392167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 316.401485] Lustre: Skipped 2 previous similar messages [ 321.548118] Lustre: server umount lustre-OST0000 complete [ 326.627429] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 326.637097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 326.639757] Lustre: Skipped 1 previous similar message [ 326.649668] Lustre: Skipped 2 previous similar messages [ 330.309777] Lustre: server umount lustre-MDT0000 complete [ 331.744222] LustreError: 15056:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 331.756071] LustreError: 15056:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 332.572393] LustreError: 17010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789264293 with bad export cookie 14641862161375980398 [ 332.573797] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 332.583073] LustreError: 17010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 332.795705] Lustre: server umount lustre-MDT0001 complete [ 336.373432] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 21:51:35 (1789264295) [ 350.102992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 350.551751] LustreError: 19885:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 350.624134] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 354.265732] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 355.809543] LustreError: 19886:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 361.345798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 364.866801] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 371.101250] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 375.250246] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 376.690410] Lustre: Failing over lustre-MDT0000 [ 376.900982] Lustre: server umount lustre-MDT0000 complete [ 377.313175] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 377.317127] LustreError: Skipped 1 previous similar message [ 377.320346] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 377.333107] LustreError: 19885:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 377.342427] LustreError: 19885:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 393.164488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 393.252864] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 397.006229] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 398.816850] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 398.832319] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 398.852538] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 404.898536] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 409.057609] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 409.073207] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 409.097364] Lustre: Skipped 1 previous similar message [ 409.102457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 414.582048] Lustre: server umount lustre-MDT0000 complete [ 417.730866] LustreError: 19866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789264378 with bad export cookie 14641862161375981819 [ 417.731560] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 417.744648] LustreError: 19866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 418.010582] Lustre: server umount lustre-MDT0001 complete [ 419.362085] hrtimer: interrupt took 2090410 ns [ 424.215488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 424.551764] LustreError: 23220:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 424.567655] LustreError: 23220:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 424.613617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 424.619834] Lustre: Skipped 2 previous similar messages [ 428.234882] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 435.496851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 439.864588] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 446.664909] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 451.587712] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 453.305216] Lustre: Failing over lustre-MDT0000 [ 453.567682] Lustre: server umount lustre-MDT0000 complete [ 454.624966] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 454.632627] Lustre: Skipped 2 previous similar messages [ 459.745136] LustreError: 23215:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 459.762547] LustreError: 23215:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 471.009168] Lustre: 3629:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789264415/real 1789264415] req@ffff893a47618a80 x1876179302908928/t0(0) o400->MGC192.168.203.114@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789264431 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 471.039592] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 471.534770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 484.409218] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 485.859975] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 485.877197] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 485.882876] Lustre: Skipped 1 previous similar message [ 485.914596] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 492.538641] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 496.100026] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 496.108047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 496.114451] Lustre: Skipped 3 previous similar messages [ 502.133816] Lustre: server umount lustre-MDT0000 complete [ 505.019080] LustreError: 23201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789264465 with bad export cookie 14641862161375982757 [ 505.023883] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 505.032771] LustreError: 23201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 505.353323] Lustre: server umount lustre-MDT0001 complete [ 512.388415] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 21:54:31 (1789264471) [ 525.166944] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 532.564981] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 560.532740] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 568.485562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 568.644752] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 568.668664] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 568.724571] Lustre: lustre-MDT0000: new disk, initializing [ 568.787965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 568.795578] Lustre: Skipped 2 previous similar messages [ 568.815827] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 571.766852] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 579.626454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 579.731218] Lustre: 29584:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 579.975272] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 583.008850] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 589.448262] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 592.736504] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 596.968403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 597.146477] Lustre: 31551:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 598.373689] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 598.397635] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 598.422984] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 598.430849] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 600.653293] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 606.383508] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 611.699261] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 624.098218] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 624.103673] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 624.106320] LustreError: Skipped 1 previous similar message [ 624.108276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 624.108282] Lustre: Skipped 2 previous similar messages [ 624.127406] Lustre: Skipped 3 previous similar messages [ 629.603370] Lustre: server umount lustre-OST0000 complete [ 638.586370] Lustre: server umount lustre-MDT0000 complete [ 639.457883] LustreError: 30473:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 639.469943] LustreError: 30473:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 641.224194] LustreError: 29579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789264602 with bad export cookie 14641862161375983163 [ 641.228896] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 641.232435] LustreError: 29579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 641.560652] Lustre: server umount lustre-MDT0001 complete [ 648.919666] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 651.049835] Key type lgssc unregistered [ 651.288665] LNet: 33525:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 651.296422] LNetError: 33525:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 651.319217] LNet: Removed LNI 192.168.203.114@tcp [ 652.027358] Key type .llcrypt unregistered [ 652.030527] Key type ._llcrypt unregistered [ 668.586532] Key type ._llcrypt registered [ 668.589730] Key type .llcrypt registered [ 668.655896] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 669.100523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 669.149307] alg: No test for adler32 (adler32-zlib) [ 670.155892] Lustre: Lustre: Build Version: 2.17.57_104_g543d5f9 [ 670.347824] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 671.967196] Key type lgssc registered [ 672.697840] Lustre: Echo OBD driver; http://www.lustre.org/ [ 677.854627] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 677.868732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 679.157052] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 681.748951] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 687.803546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 688.039454] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 691.435777] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 696.946866] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 701.037374] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 706.506494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 706.747597] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 711.687447] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 713.903756] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 717.500699] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 736.742896] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 21:58:15 (1789264695) [ 738.314402] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 739.828653] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 744.759679] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 21:58:23 (1789264703) [ 751.326703] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 21:58:30 (1789264710) [ 752.056884] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 758.165975] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 758.175705] Lustre: Skipped 2 previous similar messages [ 759.410536] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 21:58:38 (1789264718) [ 762.350769] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 762.354184] Lustre: Skipped 3 previous similar messages [ 775.016096] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 775.021825] Lustre: Skipped 6 previous similar messages [ 776.872803] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 21:58:56 (1789264736) [ 779.087845] Lustre: *** cfs_fail_loc=131b, val=2*** [ 781.722441] Lustre: *** cfs_fail_loc=131b, val=2*** [ 787.944130] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 21:59:07 (1789264747) [ 791.221544] Lustre: Setting parameter general.jobid_name=TESTNAME1 in log params [ 791.229485] Lustre: Skipped 5 previous similar messages [ 798.765881] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 21:59:17 (1789264757) [ 811.491254] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 811.492401] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 811.503730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 811.511703] Lustre: Skipped 1 previous similar message [ 816.613856] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 816.617441] Lustre: Skipped 2 previous similar messages [ 821.729799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 821.735753] Lustre: Skipped 1 previous similar message [ 823.264626] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 823.411991] Lustre: server umount lustre-OST0000 complete [ 826.851330] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 826.866082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 831.457701] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 831.464521] LustreError: Skipped 1 previous similar message [ 831.469380] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 831.481346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 832.686334] Lustre: server umount lustre-MDT0000 complete [ 835.445071] LustreError: 34894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789264796 with bad export cookie 6568096810331006465 [ 835.451314] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 835.461335] LustreError: 34894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 835.792333] Lustre: server umount lustre-MDT0001 complete [ 843.836441] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 845.990827] Key type lgssc unregistered [ 846.202556] LNet: 42771:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 846.207648] LNetError: 42771:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 846.218977] LNet: Removed LNI 192.168.203.114@tcp [ 846.853386] Key type .llcrypt unregistered [ 846.856196] Key type ._llcrypt unregistered [ 860.028796] Key type ._llcrypt registered [ 860.030954] Key type .llcrypt registered [ 860.102656] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 870.640975] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 871.197849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 871.207636] alg: No test for adler32 (adler32-zlib) [ 872.146758] Lustre: Lustre: Build Version: 2.17.57_104_g543d5f9 [ 872.335735] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 873.999629] Key type lgssc registered [ 874.797499] Lustre: Echo OBD driver; http://www.lustre.org/ [ 900.629941] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 900.654506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 901.960302] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 902.000581] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 902.080914] Lustre: lustre-MDT0000: new disk, initializing [ 902.131531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 902.144828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 905.310860] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 913.499213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 913.586793] Lustre: 45911:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 913.631087] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 913.634549] Lustre: Skipped 1 previous similar message [ 913.720667] Lustre: lustre-MDT0001: new disk, initializing [ 913.817319] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 913.864717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 913.883501] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 916.814504] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 923.845469] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 927.811054] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 932.266133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 932.461125] Lustre: lustre-OST0000: new disk, initializing [ 932.467347] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 932.474593] Lustre: 47878:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 932.538110] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 936.718987] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 938.526273] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 938.534743] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 938.601712] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 943.954713] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 946.927957] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 947.093734] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 950.314542] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 950.490675] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 953.825594] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 953.830327] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 953.833502] LustreError: Skipped 1 previous similar message [ 953.843977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 953.848305] Lustre: Skipped 1 previous similar message [ 957.823803] Lustre: server umount lustre-OST0000 complete [ 964.064342] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 964.068712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 964.073618] Lustre: Skipped 2 previous similar messages [ 964.080055] Lustre: Skipped 1 previous similar message [ 969.184799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 969.187590] Lustre: Skipped 1 previous similar message [ 973.281215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 973.285722] Lustre: Skipped 1 previous similar message [ 974.303099] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 974.417762] Lustre: server umount lustre-MDT0000 complete [ 976.637189] LustreError: 45890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789264937 with bad export cookie 13091079345222420824 [ 976.641669] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 976.647790] LustreError: 45890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 976.833855] Lustre: server umount lustre-MDT0001 complete [ 982.185158] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 22:02:21 (1789264941) [ 986.949443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 987.555839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 991.538827] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 992.813306] LustreError: 50341:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 992.829957] LustreError: 50341:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 997.856090] LustreError: 50340:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 998.893832] LustreError: 50341:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1004.005335] LustreError: 50340:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1004.126385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1004.886236] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1009.965303] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1018.998154] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1026.187329] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1036.008977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1036.574779] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1046.417615] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1060.207946] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1067.534450] LustreError: 52873:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1068.978371] Lustre: Modifying parameter general.timeout=1 in log params [ 1070.391860] Lustre: Modifying parameter general.timeout=2 in log params [ 1073.259314] Lustre: Modifying parameter general.timeout=5 in log params [ 1073.262389] Lustre: Skipped 2 previous similar messages [ 1077.960096] Lustre: Modifying parameter general.timeout=11 in log params [ 1077.963435] Lustre: Skipped 5 previous similar messages [ 1086.044361] Lustre: Modifying parameter general.timeout=21 in log params [ 1086.048394] Lustre: Skipped 9 previous similar messages [ 1102.607858] Lustre: Modifying parameter general.timeout=42 in log params [ 1102.612205] Lustre: Skipped 20 previous similar messages [ 1135.232959] Lustre: Modifying parameter general.timeout=82 in log params [ 1135.241925] Lustre: Skipped 39 previous similar messages [ 1184.283111] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 22:05:42 (1789265142) [ 1186.749442] Lustre: 59805:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1188.415213] Lustre: 59913:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1191.766548] Lustre: 60109:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1191.777394] Lustre: 60109:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1199.491272] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 in log params [ 1199.500589] Lustre: Skipped 57 previous similar messages [ 1220.135226] Lustre: 61668:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1220.143967] Lustre: 61668:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1220.166115] LustreError: 61668:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1223.732056] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 22:06:22 (1789265182) [ 1235.938819] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1235.948051] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1235.979882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1239.015463] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1239.028433] Lustre: Skipped 1 previous similar message [ 1241.418847] Lustre: server umount lustre-MDT0000 complete [ 1244.138858] LustreError: 50340:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1244.171169] LustreError: 50340:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1245.091298] LustreError: 56336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789265205 with bad export cookie 13091079345222421706 [ 1245.106936] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1245.107310] LustreError: 56336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1245.506790] Lustre: server umount lustre-MDT0001 complete [ 1260.295249] Lustre: server umount lustre-OST0000 complete [ 1287.109648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1287.375338] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1287.600319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1291.762170] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1300.284088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1300.478152] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1300.520392] Lustre: 64201:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1306.170723] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1313.914579] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1318.818689] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1325.633607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1325.803802] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1326.052521] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1326.065700] Lustre: Skipped 1 previous similar message [ 1332.116820] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1339.695090] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1341.534772] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1341.543568] Lustre: Skipped 13 previous similar messages [ 1341.580633] Lustre: 66767:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1349.602367] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1349.609165] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1349.625662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1349.631123] Lustre: Skipped 3 previous similar messages [ 1355.174644] Lustre: server umount lustre-OST0000 complete [ 1360.355278] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1360.379717] Lustre: Skipped 2 previous similar messages [ 1370.596827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1370.608018] Lustre: Skipped 7 previous similar messages [ 1374.175167] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1374.433390] Lustre: server umount lustre-MDT0000 complete [ 1375.714782] LustreError: 64212:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1375.736329] LustreError: 64212:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1378.497026] LustreError: 64942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789265339 with bad export cookie 13091079345222424667 [ 1378.505623] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1378.509649] LustreError: 64942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1378.933877] Lustre: server umount lustre-MDT0001 complete [ 1390.546548] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 1393.330330] Key type lgssc unregistered [ 1393.668609] LNet: 67974:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1393.676792] LNetError: 67974:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1393.697844] LNet: Removed LNI 192.168.203.114@tcp [ 1394.708644] Key type .llcrypt unregistered [ 1394.710486] Key type ._llcrypt unregistered [ 1422.233798] Key type ._llcrypt registered [ 1422.236444] Key type .llcrypt registered [ 1422.341389] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 1423.295110] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1423.466098] alg: No test for adler32 (adler32-zlib) [ 1424.547817] Lustre: Lustre: Build Version: 2.17.57_104_g543d5f9 [ 1424.803415] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 1426.471351] Key type lgssc registered [ 1427.602445] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1435.699627] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1435.720545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1437.375760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1442.999817] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1447.905650] LustreError: 69384:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1447.934785] LustreError: 69384:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1452.020390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1452.807476] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1457.730679] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1465.593793] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1470.716923] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1477.836754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1478.429179] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1484.109811] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1491.513313] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1503.540939] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1513.442316] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1513.452730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1513.462817] Lustre: Skipped 2 previous similar messages [ 1513.484054] Lustre: Skipped 2 previous similar messages [ 1518.563166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1518.576273] Lustre: Skipped 2 previous similar messages [ 1519.257324] Lustre: server umount lustre-MDT0000 complete [ 1523.235732] LustreError: 69365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789265484 with bad export cookie 1916886859241845106 [ 1523.240631] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1523.251778] LustreError: 69365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1523.629309] Lustre: server umount lustre-MDT0001 complete [ 1538.372545] Lustre: server umount lustre-OST0000 complete [ 1565.484454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1565.918311] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1565.980736] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1566.261636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1570.970804] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1580.123456] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1580.236908] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1580.272240] Lustre: 74256:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1580.717618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1584.993473] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1592.614130] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1598.213831] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1605.492450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1605.704488] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1606.096111] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1611.733353] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1618.855850] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1620.599375] Lustre: 76832:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1622.209544] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1630.688984] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1630.701102] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1630.711717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1631.214626] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1633.564894] Lustre: server umount lustre-OST0000 complete [ 1641.452127] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1641.455520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1641.472388] Lustre: Skipped 2 previous similar messages [ 1641.477168] Lustre: Skipped 2 previous similar messages [ 1643.292705] Lustre: server umount lustre-MDT0000 complete [ 1646.566975] LustreError: 74263:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1646.582940] LustreError: 74263:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1646.685942] LustreError: 74249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789265607 with bad export cookie 1916886859241845946 [ 1646.690135] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1646.694343] LustreError: 74249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1647.023491] Lustre: server umount lustre-MDT0001 complete [ 1658.780063] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 1661.366427] Key type lgssc unregistered [ 1661.656498] LNet: 78043:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1661.664702] LNetError: 78043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1661.684117] LNet: Removed LNI 192.168.203.114@tcp [ 1662.547159] Key type .llcrypt unregistered [ 1662.553525] Key type ._llcrypt unregistered [ 1687.834721] Key type ._llcrypt registered [ 1687.836521] Key type .llcrypt registered [ 1687.960195] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 1689.402806] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1689.474325] alg: No test for adler32 (adler32-zlib) [ 1690.748305] Lustre: Lustre: Build Version: 2.17.57_104_g543d5f9 [ 1691.063275] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 1692.711246] Key type lgssc registered [ 1694.362563] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1703.943599] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1703.975389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1705.892441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1710.850265] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1716.197696] LustreError: 79467:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1716.217620] LustreError: 79467:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1721.321823] LustreError: 79468:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1722.768925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1723.882629] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1730.350189] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1737.031519] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1743.507482] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1750.821860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1751.405130] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1757.646381] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1765.273397] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1780.334093] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1781.334480] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1783.119645] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1783.130134] Lustre: Skipped 1 previous similar message [ 1785.492825] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1785.498517] Lustre: Skipped 2 previous similar messages [ 1790.206738] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1790.211083] Lustre: Skipped 4 previous similar messages [ 1798.213625] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1798.219217] Lustre: Skipped 9 previous similar messages [ 1815.202736] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1815.214576] Lustre: Skipped 18 previous similar messages [ 1847.812333] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1847.820282] Lustre: Skipped 34 previous similar messages [ 1880.220768] Lustre: 87595:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1883.560494] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 22:17:21 (1789265841) [ 1885.333451] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1887.325444] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 22:17:25 (1789265845) [ 1891.811871] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1891.821123] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1891.857160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1893.860829] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1893.863981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1893.885121] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1893.915144] Lustre: Skipped 1 previous similar message [ 1897.749167] Lustre: server umount lustre-OST0000 complete [ 1902.571457] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1902.586893] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1902.600673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1909.217130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1909.224837] Lustre: Skipped 3 previous similar messages [ 1914.337500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1914.343665] Lustre: Skipped 1 previous similar message [ 1916.383140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1916.547969] Lustre: server umount lustre-MDT0000 complete [ 1918.437471] LustreError: 79468:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1918.450452] LustreError: 79468:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1920.241627] LustreError: 79447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789265881 with bad export cookie 4740903453417216456 [ 1920.246024] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1920.255782] LustreError: 79447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1920.608255] Lustre: server umount lustre-MDT0001 complete [ 1932.308872] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 1935.135836] Key type lgssc unregistered [ 1935.371773] LNet: 88844:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1935.383566] LNetError: 88844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1935.411077] LNet: Removed LNI 192.168.203.114@tcp [ 1936.451093] Key type .llcrypt unregistered [ 1936.454292] Key type ._llcrypt unregistered [ 1940.780588] Key type ._llcrypt registered [ 1940.784114] Key type .llcrypt registered [ 1940.878211] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1946.821598] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules [ 1948.167538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1948.241821] alg: No test for adler32 (adler32-zlib) [ 1949.461802] Lustre: Lustre: Build Version: 2.17.57_104_g543d5f9 [ 1949.791692] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 1951.555688] Key type lgssc registered [ 1952.860914] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1954.162322] LustreError: 90091:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 1971.690173] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 1976.431395] LustreError: 90091:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1980.799918] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1980.844062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1982.334482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1986.942448] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 1992.675582] LustreError: 91116:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1992.689750] LustreError: 91116:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1995.731892] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 22:19:14 (1789265954) [ 1997.795108] LustreError: 91117:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2000.230645] LustreError: 91148:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2000.255659] LustreError: 91148:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2000.264658] LustreError: 91148:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 2000.681392] Lustre: server umount lustre-MDT0000 complete [ 2014.244347] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 2017.426588] Key type lgssc unregistered [ 2017.714651] LNet: 92642:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2017.727292] LNetError: 92642:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2017.772610] LNet: Removed LNI 192.168.203.114@tcp [ 2018.410117] Key type .llcrypt unregistered [ 2018.421091] Key type ._llcrypt unregistered [ 2042.168237] Key type ._llcrypt registered [ 2042.170958] Key type .llcrypt registered [ 2042.263203] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 2043.327295] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2043.359663] alg: No test for adler32 (adler32-zlib) [ 2044.369750] Lustre: Lustre: Build Version: 2.17.57_104_g543d5f9 [ 2044.669988] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 2046.360617] Key type lgssc registered [ 2047.417971] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2055.263171] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2055.303150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2057.037152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2061.955995] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2066.403503] LustreError: 93858:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2066.418384] LustreError: 93858:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2071.525634] LustreError: 93857:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2072.069873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2072.891295] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2078.133323] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2085.813662] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2091.725111] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2098.164178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2098.928633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2104.159445] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2110.370769] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2159.551560] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 22:21:58 (1789266118) [ 2161.026216] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2162.739034] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 22:22:01 (1789266121) [ 2168.805512] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2168.807672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2168.822674] Lustre: Skipped 2 previous similar messages [ 2168.851820] Lustre: Skipped 2 previous similar messages [ 2172.725916] Lustre: server umount lustre-MDT0000 complete [ 2173.927319] LustreError: 93852:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2173.940912] LustreError: 93852:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2176.171260] LustreError: 93837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789266136 with bad export cookie 10870398521273567921 [ 2176.175213] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2176.181317] LustreError: 93837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2176.500468] Lustre: server umount lustre-MDT0001 complete [ 2189.511125] Lustre: server umount lustre-OST0000 complete [ 2199.778745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2200.336423] LustreError: 98131:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2200.592648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2205.434254] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2205.673278] LustreError: 98132:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2215.119125] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2216.044502] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2221.550204] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2229.583874] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2235.183742] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2249.288251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2249.543495] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2249.570490] LustreError: 98120:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2249.637797] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2249.665246] LustreError: 100214:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2249.668861] LustreError: 100214:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2249.674584] LustreError: 100214:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2249.779684] Lustre: server umount lustre-OST0000 complete [ 2249.803200] LustreError: 100214:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2259.852389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2260.040986] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2260.048572] LustreError: 98121:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2260.068362] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2260.085141] LustreError: 100681:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2260.093094] LustreError: 100681:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2260.099265] LustreError: 100681:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2260.148671] Lustre: server umount lustre-OST0000 complete [ 2260.151601] LustreError: 100681:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2269.641089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2269.852339] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2269.862129] Lustre: Found index 0 for lustre-OST0000, updating log [ 2269.872470] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2269.968403] Lustre: lustre-OST0000: new disk, initializing [ 2269.977428] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2269.988864] Lustre: 101216:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2270.243690] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2276.257154] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2279.977748] Lustre: Failing over lustre-OST0000 [ 2280.070475] Lustre: server umount lustre-OST0000 complete [ 2285.287875] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2285.302977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2285.324905] Lustre: Skipped 1 previous similar message [ 2285.340845] Lustre: Skipped 1 previous similar message [ 2290.657387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2290.665388] Lustre: Skipped 1 previous similar message [ 2293.836354] Lustre: server umount lustre-MDT0000 complete [ 2295.781452] LustreError: 99118:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2295.799216] LustreError: 99118:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2297.734875] LustreError: 98112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789266258 with bad export cookie 10870398521273570665 [ 2297.737671] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2297.751307] LustreError: 98112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2298.102299] Lustre: server umount lustre-MDT0001 complete [ 2309.453780] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 22:24:27 (1789266267) [ 2330.853507] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 2343.157059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2343.462589] LustreError: 103564:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2343.545265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2348.258774] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2358.374269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2365.616883] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2369.572220] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2378.841961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2379.899630] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2379.907814] Lustre: Skipped 1 previous similar message [ 2379.932594] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2383.986658] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2384.007163] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2384.050725] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2388.536404] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2389.115707] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:04 [ 2389.139181] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2389.142524] Lustre: lustre-OST0000: Recovery over after 0:06, of 1 clients 1 recovered and 0 were evicted. [ 2389.156177] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2389.174796] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2398.684093] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2398.788935] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2398.913939] Lustre: lustre-OST0001: new disk, initializing [ 2398.917868] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2398.923670] Lustre: 105887:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2406.067046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2406.086682] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2406.275544] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2408.196249] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2417.369324] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2428.844852] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2436.585868] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2436.595781] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2436.597253] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.14@tcp (stopping) [ 2436.626633] Lustre: Skipped 1 previous similar message [ 2441.692118] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.14@tcp (stopping) [ 2441.695295] Lustre: Skipped 3 previous similar messages [ 2447.330060] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2447.577897] Lustre: server umount lustre-MDT0001 complete [ 2451.907596] LustreError: 103560:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2451.917505] LustreError: 103560:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2458.279877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2458.387748] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2458.408073] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2458.480673] Lustre: 103552:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2459.492364] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2459.495075] Lustre: Skipped 1 previous similar message [ 2460.358106] LustreError: 107431:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2460.381292] LustreError: 107431:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2460.406356] LustreError: 103556:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2460.567884] LustreError: 107442:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.114@tcp: cfg command failed: rc = -17 [ 2460.584934] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2460.584934] [ 2464.646781] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2464.763291] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2475.116995] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 22:27:13 (1789266433) [ 2494.830783] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 2514.651693] Lustre: 109502:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2538.823350] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2542.543907] Lustre: 110638:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2628.064091] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2628.070111] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2628.084555] Lustre: Skipped 2 previous similar messages [ 2628.095823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2628.097979] Lustre: Skipped 7 previous similar messages [ 2633.050043] Lustre: server umount lustre-MDT0000 complete [ 2633.723603] LustreError: 106157:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2633.753446] LustreError: 106157:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 2637.306551] LustreError: 103547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789266598 with bad export cookie 10870398521273571666 [ 2637.309736] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2637.312135] LustreError: 103547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2637.676962] Lustre: server umount lustre-MDT0001 complete [ 2652.894803] Lustre: server umount lustre-OST0000 complete [ 2654.689103] Lustre: 93168:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789266599/real 1789266599] req@ffff893a50fc7800 x1876181390826880/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789266615 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 2654.732693] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2654.753795] Lustre: Skipped 3 previous similar messages [ 2657.606634] Lustre: server umount lustre-OST0001 complete [ 2664.890599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2678.929498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2688.884206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2700.035915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2722.279400] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 2735.891404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2735.910560] Lustre: lustre-MDT0000: reset Object Index mappings [ 2736.862616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2742.899629] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2752.716678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2752.748651] Lustre: lustre-MDT0001: reset Object Index mappings [ 2758.729300] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2762.152210] Lustre: 115738:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2768.440286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2774.752068] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2777.268791] LustreError: 116093:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2777.299485] LustreError: 116093:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2777.320881] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 2782.397150] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:418) [ 2783.775583] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2789.897288] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 2789.908475] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 2791.333175] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 2800.016593] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2805.180682] Lustre: 117615:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2865.701698] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 22:33:44 (1789266824) [ 2891.232458] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2891.237065] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2891.258051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2891.268196] Lustre: Skipped 3 previous similar messages [ 2897.368189] Lustre: server umount lustre-MDT0000 complete [ 2901.488611] LustreError: 114573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789266862 with bad export cookie 10870398521273618741 [ 2901.503574] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2901.504739] LustreError: 114573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2908.359549] Lustre: server umount lustre-MDT0001 complete [ 2919.349949] Lustre: server umount lustre-OST0000 complete [ 2938.335143] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2938.616881] Lustre: server umount lustre-OST0001 complete [ 2946.821500] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 2958.035088] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 3010.138176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3010.482447] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3010.529856] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3010.743643] Lustre: lustre-MDT0000: new disk, initializing [ 3010.854439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3010.867521] Lustre: Skipped 3 previous similar messages [ 3010.903651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3015.996493] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3024.651832] Lustre: server umount lustre-MDT0000 complete [ 3037.849925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3038.372414] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3038.408569] Lustre: 121895:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3038.440198] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3044.263652] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3049.923239] Lustre: server umount lustre-MDT0000 complete [ 3060.493267] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 22:36:58 (1789267018) [ 3062.398285] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3064.670239] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 22:37:02 (1789267022) [ 3083.963209] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 3094.363416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3099.732276] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3109.069659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3109.256245] Lustre: 123945:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3109.335668] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3109.343771] Lustre: Skipped 1 previous similar message [ 3109.491552] Lustre: lustre-MDT0001: new disk, initializing [ 3109.627907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3109.654665] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3114.673447] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3119.529423] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3126.242918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3126.508916] Lustre: lustre-OST0000: new disk, initializing [ 3126.518336] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3126.525683] Lustre: 125522:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3128.144407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3128.162701] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3128.311474] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3132.720623] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3143.777067] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3143.941762] Lustre: lustre-OST0001: new disk, initializing [ 3143.963689] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3143.975753] Lustre: 126394:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3145.798195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3145.816662] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3145.941760] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3150.100793] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3160.130939] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3163.709329] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3557.393137] Lustre: 132810:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3557.401279] Lustre: 132810:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3557.406361] Lustre: 132810:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3557.417600] Lustre: 132810:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3557.425471] Lustre: 132810:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3557.430623] Lustre: 132810:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3849.580476] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 22:50:07 (1789267807) [ 3867.621013] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3867.631622] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3867.648374] Lustre: Skipped 5 previous similar messages [ 3867.653687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3867.659420] Lustre: Skipped 8 previous similar messages [ 3870.665287] Lustre: server umount lustre-MDT0000 complete [ 3873.251299] LustreError: 127197:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3873.271382] LustreError: 127197:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 3873.851174] LustreError: 132833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789267834 with bad export cookie 10870398521273689371 [ 3873.861162] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3873.867067] LustreError: 132833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3878.376350] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3878.389433] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3878.395725] Lustre: Skipped 4 previous similar messages [ 3878.409415] Lustre: Skipped 5 previous similar messages [ 3880.388611] Lustre: server umount lustre-MDT0001 complete [ 3898.337920] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3898.459545] Lustre: server umount lustre-OST0000 complete [ 3908.306874] Lustre: server umount lustre-OST0001 complete [ 3915.376777] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 3927.742812] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 3973.269212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3973.432820] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3973.453739] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3973.531170] Lustre: lustre-MDT0000: new disk, initializing [ 3973.626811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3973.631056] Lustre: Skipped 5 previous similar messages [ 3973.652659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3978.314696] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 3988.388635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3988.480260] Lustre: 135872:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3988.524586] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3988.529072] Lustre: Skipped 1 previous similar message [ 3988.593868] Lustre: lustre-MDT0001: new disk, initializing [ 3988.701526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3988.713996] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3993.760806] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4003.655344] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4010.281050] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4018.625782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4018.984422] Lustre: lustre-OST0000: new disk, initializing [ 4018.988410] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4018.997982] Lustre: 137842:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4020.633646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4020.650822] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4020.796100] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4026.126667] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 4036.885334] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4046.787456] Lustre: lustre-MDD0000: changelog on [ 4049.692421] Lustre: lustre-MDD0001: changelog on [ 6077.550200] Lustre: lustre-MDD0001: changelog off [ 6079.664618] Lustre: 135878:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6080.758536] Lustre: lustre-MDD0000: changelog off [ 6089.189592] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6089.191886] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6089.206769] LustreError: Skipped 1 previous similar message [ 6089.208546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6089.222027] Lustre: Skipped 1 previous similar message [ 6089.247595] Lustre: Skipped 1 previous similar message [ 6090.212156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6090.222897] Lustre: Skipped 1 previous similar message [ 6093.499394] Lustre: server umount lustre-OST0000 complete [ 6098.404371] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6098.411039] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6098.443301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6100.461768] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6104.596265] Lustre: server umount lustre-MDT0000 complete [ 6105.573177] LustreError: 135885:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6105.590676] LustreError: 135885:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6108.111104] LustreError: 135865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789270068 with bad export cookie 10870398521274512522 [ 6108.128502] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6108.132523] LustreError: 135865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6108.554165] Lustre: server umount lustre-MDT0001 complete [ 6120.047194] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 6123.084475] Key type lgssc unregistered [ 6123.469946] LNet: 147003:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6123.482061] LNetError: 147003:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6123.502448] LNet: Removed LNI 192.168.203.114@tcp [ 6124.469171] Key type .llcrypt unregistered [ 6124.474763] Key type ._llcrypt unregistered [ 6145.953977] Key type ._llcrypt registered [ 6145.958025] Key type .llcrypt registered [ 6146.136862] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 6162.253754] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 6163.869665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6163.914901] alg: No test for adler32 (adler32-zlib) [ 6165.071161] Lustre: Lustre: Build Version: 2.17.57_104_g543d5f9 [ 6165.382727] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 6167.087173] Key type lgssc registered [ 6168.532661] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6208.344873] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6208.387669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6209.688075] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6209.716240] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6209.789504] Lustre: lustre-MDT0000: new disk, initializing [ 6209.897581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6209.912174] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6215.009644] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6226.411433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6226.572304] Lustre: 150201:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6226.624604] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6226.630223] Lustre: Skipped 1 previous similar message [ 6226.732531] Lustre: lustre-MDT0001: new disk, initializing [ 6226.844110] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6226.875058] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6226.881996] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6232.237207] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6241.775482] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6248.047376] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6254.853571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6255.114135] Lustre: lustre-OST0000: new disk, initializing [ 6255.127415] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6255.134580] Lustre: 152172:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6255.209106] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6261.823917] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6263.412472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6263.435529] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6263.613148] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6272.352322] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6278.341278] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_module obdecho/obdecho [ 6280.332790] LustreError: 153174:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6280.340253] LustreError: 153174:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6282.449822] LustreError: 153271:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6298.606364] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6298.619608] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6298.629556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6299.618190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6299.620990] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6301.232358] Lustre: server umount lustre-MDT0000 complete [ 6305.563158] LustreError: 150193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789270266 with bad export cookie 15717219989975574525 [ 6305.568326] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6305.577963] LustreError: 150193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6306.171528] Lustre: server umount lustre-MDT0001 complete [ 6321.203723] Lustre: server umount lustre-OST0000 complete [ 6344.772936] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 6358.070672] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 6399.401283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6399.693126] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6399.736984] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6399.797884] Lustre: lustre-MDT0000: new disk, initializing [ 6399.883275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6399.926733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6404.279159] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6414.983045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6415.122643] Lustre: 156988:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6415.193488] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6415.196966] Lustre: Skipped 1 previous similar message [ 6415.355389] Lustre: lustre-MDT0001: new disk, initializing [ 6415.518477] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6415.552705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6415.566749] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6420.325524] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6429.885685] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6435.495608] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6442.550166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6442.797489] Lustre: lustre-OST0000: new disk, initializing [ 6442.801206] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6442.806244] Lustre: 158956:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6442.889944] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6448.209058] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6448.225664] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6448.336786] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6448.942395] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6457.796434] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6462.647605] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6463.020714] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6468.240730] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6468.455917] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6474.208401] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6474.211634] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6474.226283] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6474.237961] Lustre: Skipped 2 previous similar messages [ 6474.250336] Lustre: Skipped 3 previous similar messages [ 6476.599971] Lustre: server umount lustre-OST0000 complete [ 6482.400433] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6482.411653] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6482.430169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6486.452332] Lustre: server umount lustre-MDT0000 complete [ 6489.577245] LustreError: 156999:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6489.598851] LustreError: 156999:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6489.733925] LustreError: 158955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789270450 with bad export cookie 15717219989975575484 [ 6489.740225] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6489.748664] LustreError: 158955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6490.133211] Lustre: server umount lustre-MDT0001 complete [ 6493.361678] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 23:34:12 (1789270452) [ 6494.784597] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6496.443934] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 23:34:15 (1789270455) [ 6503.484179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6503.929277] LustreError: 161368:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6503.995520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6508.629350] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6509.026888] LustreError: 161369:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6514.150458] LustreError: 161368:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6515.917469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6519.912472] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6527.565365] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6533.961713] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6540.998507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6541.235982] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6541.244417] Lustre: Skipped 1 previous similar message [ 6547.490876] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6556.281278] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6567.386548] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.14@tcp (stopping) [ 6567.392350] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6567.392373] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6567.392376] Lustre: Skipped 1 previous similar message [ 6567.414018] Lustre: Skipped 3 previous similar messages [ 6568.258839] Lustre: server umount lustre-MDT0000 complete [ 6570.978094] LustreError: 161368:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6571.002149] LustreError: 161368:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6572.128535] LustreError: 162104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789270532 with bad export cookie 15717219989975576366 [ 6572.131752] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6572.144256] LustreError: 162104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6576.096528] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6576.110210] Lustre: Skipped 2 previous similar messages [ 6576.120028] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6576.138673] Lustre: Skipped 1 previous similar message [ 6578.712557] Lustre: server umount lustre-MDT0001 complete [ 6585.073982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6598.287517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6598.665145] LustreError: 165210:0:(llog.c:1655:llog_backup()) MGC192.168.203.114@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6598.672117] Lustre: 165210:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.114@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6601.696472] LustreError: 165215:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6601.722588] LustreError: 165215:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6618.560431] LustreError: 165216:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6618.599182] LustreError: 165216:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 6651.872871] LustreError: 163199:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6651.899964] LustreError: 163199:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 25 previous similar messages [ 6659.551507] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6659.559816] Lustre: 165210:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.114@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6700.558171] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6705.090805] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6705.657240] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6714.177195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6715.872564] LustreError: 165215:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6715.915191] LustreError: 165215:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 44 previous similar messages [ 6718.107570] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -11 [ 6722.641969] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6723.573350] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6728.910413] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6730.335701] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6736.174684] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6737.774806] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6759.398058] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6759.401885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6759.403723] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6759.419054] Lustre: Skipped 2 previous similar messages [ 6759.440250] Lustre: Skipped 4 previous similar messages [ 6760.237929] Lustre: server umount lustre-MDT0000 complete [ 6764.464131] LustreError: 161349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789270725 with bad export cookie 15717219989975582820 [ 6764.472485] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6764.491843] LustreError: 161349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6764.995955] Lustre: server umount lustre-MDT0001 complete [ 6772.437934] Lustre: server umount lustre-OST0000 complete [ 6800.565319] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 6813.118641] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 6851.199582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6851.449906] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6851.477186] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6851.588294] Lustre: lustre-MDT0000: new disk, initializing [ 6851.673746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6851.685421] Lustre: Skipped 1 previous similar message [ 6851.707595] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6856.451606] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6867.267777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6867.377747] Lustre: 170312:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6867.598486] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6872.484695] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6883.073571] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6890.542645] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6898.453230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6898.879458] Lustre: lustre-OST0000: new disk, initializing [ 6898.882707] Lustre: Skipped 1 previous similar message [ 6898.897206] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6898.911562] Lustre: Skipped 2 previous similar messages [ 6898.917473] Lustre: 172281:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6907.508343] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6907.521957] Lustre: Skipped 1 previous similar message [ 6907.529935] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6907.687729] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6908.375300] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6918.771672] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6924.116943] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6924.442944] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6929.721359] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6929.930800] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6933.480778] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6933.488818] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6933.516964] Lustre: Skipped 1 previous similar message [ 6933.525798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6933.532657] Lustre: Skipped 2 previous similar messages [ 6946.271322] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6946.556976] Lustre: server umount lustre-OST0000 complete [ 6965.215698] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6965.443286] Lustre: server umount lustre-MDT0000 complete [ 6969.105719] LustreError: 170304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789270929 with bad export cookie 15717219989975599655 [ 6969.112643] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6969.122301] LustreError: 170304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6969.529367] Lustre: server umount lustre-MDT0001 complete [ 6978.386359] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 23:42:17 (1789270937) [ 6984.879372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6985.346842] LustreError: 174694:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6985.365177] LustreError: 174694:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6990.392586] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 6999.357359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7004.414419] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7013.833163] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7021.163504] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7032.297641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7040.802320] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7051.691813] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7069.360175] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 23:43:47 (1789271027) [ 7076.326775] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7076.329087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7076.341333] Lustre: Skipped 4 previous similar messages [ 7076.353480] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7076.357693] Lustre: Skipped 11 previous similar messages [ 7076.378609] LustreError: Skipped 1 previous similar message [ 7080.827135] Lustre: server umount lustre-OST0000 complete [ 7099.359140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7099.577899] Lustre: server umount lustre-MDT0000 complete [ 7103.311921] LustreError: 174677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789271064 with bad export cookie 15717219989975600530 [ 7103.319324] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7103.323591] LustreError: 174677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7103.662495] Lustre: server umount lustre-MDT0001 complete [ 7116.147379] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 7118.900792] Key type lgssc unregistered [ 7119.249601] LNet: 178437:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7119.257230] LNetError: 178437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7119.280238] LNet: Removed LNI 192.168.203.114@tcp [ 7120.123649] Key type .llcrypt unregistered [ 7120.129855] Key type ._llcrypt unregistered [ 7139.592322] Key type ._llcrypt registered [ 7139.597766] Key type .llcrypt registered [ 7139.670826] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 7140.757172] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7140.785327] alg: No test for adler32 (adler32-zlib) [ 7141.830897] Lustre: Lustre: Build Version: 2.17.57_104_g543d5f9 [ 7142.059407] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 7143.719236] Key type lgssc registered [ 7144.733898] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7152.009815] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7152.042659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7167.647841] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7172.768167] LustreError: 179677:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.114@tcp: failed processing log, type 4: rc = -110 [ 7177.896880] LustreError: 179677:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7177.920520] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7177.937547] LustreError: MGC192.168.203.114@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7177.969631] LustreError: 179677:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7177.986701] LustreError: 179677:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -22 [ 7177.997043] LustreError: 179677:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 7178.124632] Lustre: server umount lustre-OST0000 complete [ 7178.133407] LustreError: 179677:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7187.450554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7189.026525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7193.877845] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7199.201308] LustreError: 180176:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7199.230073] LustreError: 180176:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7202.407745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7202.878062] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7207.583526] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7215.212157] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7220.744812] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7227.542254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7227.746815] LustreError: 181993:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7227.754412] Lustre: 181993:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.203.114@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7227.937366] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7234.046504] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7241.563763] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7243.744605] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7243.758488] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7243.775166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7245.281254] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7245.282657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7245.316071] Lustre: Skipped 2 previous similar messages [ 7247.330670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7247.337051] Lustre: Skipped 2 previous similar messages [ 7249.805356] Lustre: server umount lustre-MDT0000 complete [ 7252.450157] LustreError: 180176:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7252.493995] LustreError: 180176:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7253.389683] LustreError: 182021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789271214 with bad export cookie 6105629304509990881 [ 7253.391405] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7253.402857] LustreError: 182021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7253.522110] LustreError: 180906:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000367443db x1876186735079424/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7253.696540] Lustre: server umount lustre-MDT0001 complete [ 7268.025459] Lustre: server umount lustre-OST0000 complete [ 7278.376529] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 7290.817220] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 7329.704466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7329.962813] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7330.009290] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7330.159896] Lustre: lustre-MDT0000: new disk, initializing [ 7330.242131] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7330.280218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7334.430670] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7344.935615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7345.028806] Lustre: 185521:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7345.079339] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7345.082775] Lustre: Skipped 1 previous similar message [ 7345.167331] Lustre: lustre-MDT0001: new disk, initializing [ 7345.246571] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7345.286649] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7345.297517] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7349.719315] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7358.314325] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7363.589350] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7369.794686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7370.030426] Lustre: lustre-OST0000: new disk, initializing [ 7370.034700] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7370.044537] Lustre: 187491:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7370.152919] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7376.218885] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7376.448847] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7376.459031] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7376.596816] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7385.937866] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7390.588033] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7390.810351] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7395.057784] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7395.308419] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7397.345909] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7397.347957] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7397.351320] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7397.352561] LustreError: Skipped 1 previous similar message [ 7397.373135] Lustre: Skipped 2 previous similar messages [ 7402.465584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7402.472541] Lustre: Skipped 2 previous similar messages [ 7403.343351] Lustre: server umount lustre-OST0000 complete [ 7407.588212] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7407.605979] Lustre: Skipped 1 previous similar message [ 7412.706554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7412.716155] Lustre: Skipped 3 previous similar messages [ 7412.952969] Lustre: server umount lustre-MDT0000 complete [ 7416.202735] LustreError: 186326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789271377 with bad export cookie 6105629304509991602 [ 7416.209318] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7416.216409] LustreError: 186326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7416.494431] Lustre: server umount lustre-MDT0001 complete [ 7424.675681] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 23:49:43 (1789271383) [ 7441.489191] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 7443.920564] Key type lgssc unregistered [ 7444.161649] LNet: 190433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7444.167363] LNetError: 190433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7444.179293] LNet: Removed LNI 192.168.203.114@tcp [ 7444.937392] Key type .llcrypt unregistered [ 7444.944463] Key type ._llcrypt unregistered [ 7461.763979] Key type ._llcrypt registered [ 7461.769206] Key type .llcrypt registered [ 7461.888624] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 7462.952111] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7462.980320] alg: No test for adler32 (adler32-zlib) [ 7464.224986] Lustre: Lustre: Build Version: 2.17.57_104_g543d5f9 [ 7464.442577] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 7466.127243] Key type lgssc registered [ 7467.211414] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7474.562995] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7474.588574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7476.112274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7480.185142] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7485.410650] LustreError: 191648:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7485.424545] LustreError: 191648:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7488.667348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7488.963927] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7493.079637] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7500.670096] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7507.456723] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7516.226948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7516.502267] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7516.505611] LustreError: 193513:0:(llog.c:1696:llog_backup()) MGC192.168.203.114@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7516.516305] Lustre: 193513:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.203.114@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7516.757952] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7524.470376] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7532.834897] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7535.079211] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7535.090464] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7535.114101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7539.687544] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7539.694794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7539.699843] Lustre: Skipped 1 previous similar message [ 7539.726452] Lustre: Skipped 2 previous similar messages [ 7541.534904] Lustre: server umount lustre-MDT0000 complete [ 7544.802060] LustreError: 191647:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7544.822340] LustreError: 191647:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7544.935089] LustreError: 191628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789271505 with bad export cookie 10020899217833678211 [ 7544.944972] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7544.950550] LustreError: 191628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7545.202319] Lustre: server umount lustre-MDT0001 complete [ 7559.336615] Lustre: server umount lustre-OST0000 complete [ 7567.378358] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 7576.752605] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 7609.501347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7609.880138] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7609.924138] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7610.020233] Lustre: lustre-MDT0000: new disk, initializing [ 7610.108642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7610.121735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7614.254492] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7624.355710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7624.513217] Lustre: 197038:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7624.577584] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7624.580262] Lustre: Skipped 1 previous similar message [ 7624.583693] LustreError: 197819:0:(llog.c:1696:llog_backup()) MGC192.168.203.114@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7624.589027] LustreError: 197819:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7624.596773] Lustre: 197819:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.114@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7624.628541] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7624.635146] Lustre: Skipped 1 previous similar message [ 7624.727985] Lustre: lustre-MDT0001: new disk, initializing [ 7624.823273] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7624.858904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7624.866865] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7629.016885] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7639.359861] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7645.540733] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7653.222356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7653.466569] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7653.469627] Lustre: Skipped 1 previous similar message [ 7653.471952] LustreError: 198982:0:(llog.c:1696:llog_backup()) MGC192.168.203.114@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7653.477467] LustreError: 198982:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7653.482313] Lustre: 198982:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.114@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7653.491368] Lustre: 198982:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7653.511593] Lustre: lustre-OST0000: new disk, initializing [ 7653.517379] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7653.524469] Lustre: 199005:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7653.613781] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7659.080291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7659.093078] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7659.203869] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7659.500319] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7667.871540] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7671.966876] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7672.111904] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7675.917527] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7676.189583] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7679.977955] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7679.984875] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7679.989486] LustreError: Skipped 1 previous similar message [ 7679.990973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7680.001787] Lustre: Skipped 1 previous similar message [ 7685.093866] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7685.111292] Lustre: Skipped 2 previous similar messages [ 7690.227239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7690.238113] Lustre: Skipped 1 previous similar message [ 7692.255136] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7692.419479] Lustre: server umount lustre-OST0000 complete [ 7700.450308] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7700.467257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7700.475946] Lustre: Skipped 1 previous similar message [ 7702.889288] Lustre: server umount lustre-MDT0000 complete [ 7705.574810] LustreError: 197044:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7705.594723] LustreError: 197044:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7706.610111] LustreError: 197028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789271667 with bad export cookie 10020899217833678932 [ 7706.610965] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7706.616147] LustreError: 197028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7707.018733] Lustre: server umount lustre-MDT0001 complete [ 7714.270739] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 23:54:33 (1789271673) [ 7728.001261] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 7738.096694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7738.668973] LustreError: 202134:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7738.763564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7743.971383] LustreError: 202135:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7743.992634] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7754.209590] LustreError: 202134:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7754.236059] LustreError: 202134:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7755.208977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7761.915215] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7765.419867] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7773.194076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7773.457887] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7773.463269] Lustre: Skipped 1 previous similar message [ 7780.299492] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7788.396925] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7788.562376] Lustre: lustre-OST0001: new disk, initializing [ 7788.566846] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7788.572970] Lustre: 204442:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7794.346280] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7795.818807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7795.831359] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7795.944363] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7805.248963] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7811.318967] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7813.198553] Lustre: DEBUG MARKER: ADD OST3 [ 7818.667416] loop: module loaded [ 7818.744545] loop0: detected capacity change from 0 to 204800000 [ 7818.942709] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7818.958215] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7818.989556] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7824.908979] loop0: detected capacity change from 0 to 204800000 [ 7825.935171] Lustre: DEBUG MARKER: STOP OST3 [ 7830.942630] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7831.162337] Lustre: lustre-OST0003: new disk, initializing [ 7831.166836] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7831.175682] Lustre: 206438:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7832.334797] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7832.447950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7832.460378] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7832.534675] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7834.399083] LustreError: 206562:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7835.193623] Lustre: Failing over lustre-OST0003 [ 7835.392157] Lustre: server umount lustre-OST0003 complete [ 7837.669646] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7837.684695] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7837.687152] Lustre: Skipped 1 previous similar message [ 7837.691746] LustreError: 203621:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7837.964753] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7847.412597] Lustre: DEBUG MARKER: START OST3 again [ 7855.456346] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7855.860081] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7855.872599] Lustre: Skipped 2 previous similar messages [ 7855.899478] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7857.452507] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7857.887946] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7857.888331] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7864.036519] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 7888.848174] Lustre: Failing over lustre-OST0003 [ 7888.865382] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7888.870507] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7888.881560] LustreError: Skipped 1 previous similar message [ 7888.892382] Lustre: Skipped 1 previous similar message [ 7888.893941] Lustre: lustre-OST0003: Not available for connect from 0@lo (stopping) [ 7888.903571] Lustre: Skipped 1 previous similar message [ 7889.090207] Lustre: server umount lustre-OST0003 complete [ 7891.951951] LustreError: 204152:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7891.992385] LustreError: 204152:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 7909.349287] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7909.355969] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7909.367490] Lustre: Skipped 1 previous similar message [ 7913.748695] Lustre: server umount lustre-MDT0000 complete [ 7917.225640] LustreError: 202116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789271878 with bad export cookie 10020899217833679814 [ 7917.231230] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7917.235640] LustreError: 202116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7922.660843] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7922.669470] Lustre: Skipped 8 previous similar messages [ 7923.770061] Lustre: server umount lustre-MDT0001 complete [ 7927.533921] Lustre: server umount lustre-OST0000 complete [ 7931.321430] Lustre: server umount lustre-OST0001 complete [ 7938.396686] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 7948.292896] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 7986.816864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7987.048506] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7987.092061] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7987.167409] Lustre: lustre-MDT0000: new disk, initializing [ 7987.227550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7987.247790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7992.004257] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8003.014938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8003.119754] Lustre: 211900:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8003.193791] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8003.199540] Lustre: Skipped 1 previous similar message [ 8003.315977] Lustre: lustre-MDT0001: new disk, initializing [ 8003.461640] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8003.489190] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8008.460081] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8017.989218] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8023.780709] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8030.145677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8030.348310] Lustre: 213872:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8031.743867] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8031.895622] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8037.153615] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8047.818962] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8052.821980] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8053.062509] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8058.410581] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8058.725170] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8062.432133] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8062.435905] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8062.451634] LustreError: Skipped 1 previous similar message [ 8062.460711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8062.474551] Lustre: Skipped 6 previous similar messages [ 8067.295806] Lustre: server umount lustre-OST0000 complete [ 8077.743441] Lustre: server umount lustre-MDT0000 complete [ 8077.795394] LustreError: 212768:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8077.835659] LustreError: 212768:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 8083.016252] LustreError: 212707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789272043 with bad export cookie 10020899217833682110 [ 8083.017228] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8083.022854] LustreError: 212707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8083.615476] Lustre: server umount lustre-MDT0001 complete [ 8088.365760] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 00:00:46 (1789272046) [ 8111.937649] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 8129.045587] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 8166.943518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8167.141304] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8167.168464] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8167.172187] Lustre: Skipped 1 previous similar message [ 8167.275473] Lustre: lustre-MDT0000: new disk, initializing [ 8167.284162] Lustre: Skipped 1 previous similar message [ 8167.399446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8167.409159] Lustre: Skipped 1 previous similar message [ 8171.678368] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8181.622325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8181.750891] Lustre: 218521:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8181.980934] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8186.388220] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8195.150553] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8201.073959] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8209.028618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8209.402423] Lustre: 220489:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8210.957817] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8215.626862] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8226.537281] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8232.099417] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8232.357853] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8238.062278] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8238.267577] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8241.636195] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8241.640028] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8241.646727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8241.648226] Lustre: Skipped 3 previous similar messages [ 8241.671808] Lustre: Skipped 4 previous similar messages [ 8246.470356] Lustre: server umount lustre-OST0000 complete [ 8264.160034] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8264.276491] Lustre: server umount lustre-MDT0000 complete [ 8267.232509] LustreError: 218528:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8267.249252] LustreError: 218528:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8267.662538] LustreError: 220488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789272228 with bad export cookie 10020899217833682992 [ 8267.665818] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8267.670133] LustreError: 220488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8267.961870] Lustre: server umount lustre-MDT0001 complete [ 8276.467716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8277.057013] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8277.069749] Lustre: Skipped 5 previous similar messages [ 8282.982210] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8294.527155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8300.882972] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8309.947786] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8316.549817] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8324.127179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8332.286661] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8340.674947] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8358.375052] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8358.385892] LustreError: Skipped 1 previous similar message [ 8362.372375] Lustre: server umount lustre-OST0000 complete [ 8372.967243] Lustre: server umount lustre-MDT0000 complete [ 8377.003318] LustreError: 222689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789272337 with bad export cookie 10020899217833683874 [ 8377.023286] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8377.294413] Lustre: server umount lustre-MDT0001 complete [ 8478.779530] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 8482.295511] Key type lgssc unregistered [ 8482.682904] LNet: 226391:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8482.716369] LNetError: 226391:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8482.765829] LNet: Removed LNI 192.168.203.114@tcp [ 8484.252562] Key type .llcrypt unregistered [ 8484.254286] Key type ._llcrypt unregistered [ 8518.571303] Key type ._llcrypt registered [ 8518.578551] Key type .llcrypt registered [ 8518.732338] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 8536.507686] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 8537.820965] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8537.874337] alg: No test for adler32 (adler32-zlib) [ 8538.952810] Lustre: Lustre: Build Version: 2.17.57_104_g543d5f9 [ 8539.279578] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 8540.999462] Key type lgssc registered [ 8542.616957] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8579.104180] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8579.146455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8580.427686] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8580.496649] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8580.595138] Lustre: lustre-MDT0000: new disk, initializing [ 8580.666040] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8580.701282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8584.870238] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8595.032952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8595.171683] Lustre: 229744:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8595.234264] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8595.238739] Lustre: Skipped 1 previous similar message [ 8595.330809] Lustre: lustre-MDT0001: new disk, initializing [ 8595.445234] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8595.485444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8595.502733] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8600.260526] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8609.200351] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8615.520602] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8625.961235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8626.556115] Lustre: lustre-OST0000: new disk, initializing [ 8626.579069] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8626.594232] Lustre: 231709:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8626.854757] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8635.376347] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8636.515333] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8636.530130] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8636.724095] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8647.205123] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8653.119487] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8653.326258] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8658.381499] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8658.684863] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8662.499870] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8662.505683] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8662.508913] LustreError: Skipped 1 previous similar message [ 8662.511378] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8662.521364] Lustre: Skipped 1 previous similar message [ 8662.533178] Lustre: Skipped 1 previous similar message [ 8667.640396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8667.651347] Lustre: Skipped 1 previous similar message [ 8672.737353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8672.756777] Lustre: Skipped 1 previous similar message [ 8675.301501] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8675.514336] Lustre: server umount lustre-OST0000 complete [ 8682.465872] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8682.470533] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8682.520147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8685.811293] Lustre: server umount lustre-MDT0000 complete [ 8689.101229] LustreError: 229736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789272649 with bad export cookie 12278713520800522841 [ 8689.103751] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8689.115406] LustreError: 229736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8689.394584] Lustre: server umount lustre-MDT0001 complete [ 8706.773499] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 8715.566798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8715.862720] LustreError: 234649:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8715.883155] LustreError: 234649:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8715.913556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8720.417980] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8721.389889] LustreError: 234650:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8725.475836] LustreError: 234649:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8728.840531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8729.148738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8733.455505] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8736.208257] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8742.903287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8743.259492] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8749.626269] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8759.430879] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8759.766764] Lustre: lustre-OST0001: new disk, initializing [ 8759.772149] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8759.788101] Lustre: 236958:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8759.917361] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8768.146395] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8768.750180] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8768.769212] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8768.934592] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8782.635398] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8789.442709] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8804.097120] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 00:12:42 (1789272762) [ 8809.960550] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8809.968308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8809.976970] Lustre: Skipped 4 previous similar messages [ 8810.001470] Lustre: Skipped 4 previous similar messages [ 8812.759433] Lustre: server umount lustre-MDT0000 complete [ 8815.087610] LustreError: 234645:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8815.116516] LustreError: 234645:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8816.700630] LustreError: 236957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789272777 with bad export cookie 12278713520800523723 [ 8816.706572] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8816.715635] LustreError: 236957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8817.014218] Lustre: server umount lustre-MDT0001 complete [ 8832.120069] Lustre: server umount lustre-OST0000 complete [ 8835.938886] Lustre: 227964:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789272780/real 1789272781] req@ffff893a47dc8000 x1876188201077760/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789272796 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8836.007701] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8836.827650] Lustre: server umount lustre-OST0001 complete [ 8846.451288] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 8861.367720] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 8910.440156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8911.194659] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8911.285160] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8911.467823] Lustre: lustre-MDT0000: new disk, initializing [ 8911.635614] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8911.676664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8918.674548] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8932.940177] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8933.156143] Lustre: 240928:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8933.275777] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8933.285756] Lustre: Skipped 1 previous similar message [ 8933.421625] Lustre: lustre-MDT0001: new disk, initializing [ 8933.722324] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8933.774844] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8938.893302] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8951.555596] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8960.668917] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8969.481879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8970.006746] Lustre: lustre-OST0000: new disk, initializing [ 8970.012937] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8970.034941] Lustre: 242895:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8971.339585] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8971.356713] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8971.468614] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8977.783927] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 8988.558933] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8994.718476] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8994.932363] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8999.608862] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8999.826493] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9002.477721] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9002.501284] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9002.540222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9002.549975] Lustre: Skipped 1 previous similar message [ 9016.289835] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9016.641453] Lustre: server umount lustre-OST0000 complete [ 9022.946320] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9022.965663] Lustre: Skipped 2 previous similar messages [ 9022.966539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9022.976282] Lustre: Skipped 5 previous similar messages [ 9036.255163] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9036.482446] Lustre: server umount lustre-MDT0000 complete [ 9038.312338] LustreError: 240933:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9038.349164] LustreError: 240933:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9040.791464] LustreError: 240920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789273001 with bad export cookie 12278713520800525858 [ 9040.794773] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9040.809315] LustreError: 240920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9041.235842] Lustre: server umount lustre-MDT0001 complete [ 9050.068932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9050.633032] LustreError: 245119:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9050.702237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9050.715140] Lustre: Skipped 2 previous similar messages [ 9054.944467] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 9064.194879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9071.116252] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 9080.849407] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9088.222430] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9096.401892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9096.628618] LustreError: 246942:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9096.662914] LustreError: 246942:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9104.110392] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 9113.493562] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9261.033284] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9261.040537] LustreError: Skipped 1 previous similar message [ 9261.045377] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9261.058025] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9261.061306] Lustre: Skipped 4 previous similar messages [ 9266.552250] Lustre: server umount lustre-OST0000 complete [ 9277.047259] Lustre: server umount lustre-MDT0000 complete [ 9278.434847] LustreError: 246856:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9278.469902] LustreError: 246856:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9281.348896] LustreError: 245099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789273242 with bad export cookie 12278713520800526740 [ 9281.369264] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9281.375789] LustreError: 245099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9281.705609] Lustre: server umount lustre-MDT0001 complete [ 9383.480984] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [ 9388.066903] Key type lgssc unregistered [ 9388.392606] LNet: 248730:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9388.409649] LNetError: 248730:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9388.432976] LNet: Removed LNI 192.168.203.114@tcp [ 9389.706218] Key type .llcrypt unregistered [ 9389.707935] Key type ._llcrypt unregistered [ 9422.049668] Key type ._llcrypt registered [ 9422.052246] Key type .llcrypt registered [ 9422.183427] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 9439.444803] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 9442.144170] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9442.160779] alg: No test for adler32 (adler32-zlib) [ 9443.724964] Lustre: Lustre: Build Version: 2.17.57_104_g543d5f9 [ 9444.081447] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [ 9445.743407] Key type lgssc registered [ 9447.445211] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9505.085422] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 9520.380581] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9520.421453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9521.642675] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9521.692443] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9521.764535] Lustre: lustre-MDT0000: new disk, initializing [ 9521.871825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9521.902319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9528.447636] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 9541.767607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9541.957476] Lustre: 252820:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9542.050105] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9542.058819] Lustre: Skipped 1 previous similar message [ 9542.116527] Lustre: lustre-MDT0001: new disk, initializing [ 9542.211806] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9542.263290] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9542.278962] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9547.125483] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 9552.872940] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9562.659078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9563.260535] Lustre: lustre-OST0000: new disk, initializing [ 9563.283681] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9563.302146] Lustre: 254454:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9563.313518] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 9563.561405] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9568.802337] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9568.820448] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9568.901972] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9571.831465] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 9582.555452] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9582.747796] Lustre: lustre-OST0001: new disk, initializing [ 9582.756504] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9582.769626] Lustre: 255329:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9582.962773] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9590.803510] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 9593.068192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9593.082927] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9593.173293] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9604.958608] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9613.881378] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9629.156031] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9629.163840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9629.164695] Lustre: Skipped 1 previous similar message [ 9629.166250] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9629.187950] Lustre: Skipped 3 previous similar messages [ 9633.123185] Lustre: server umount lustre-MDT0000 complete [ 9637.661460] LustreError: 252815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789273598 with bad export cookie 1704995937159613659 [ 9637.663734] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9637.688937] LustreError: 252815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9638.291849] Lustre: server umount lustre-MDT0001 complete [ 9653.964968] Lustre: server umount lustre-OST0000 complete [ 9655.775562] Lustre: 250307:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789273600/real 1789273600] req@ffff893b7d7fa680 x1876189148954624/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789273616 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9655.786141] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9655.793182] Lustre: Skipped 2 previous similar messages [ 9659.034950] Lustre: server umount lustre-OST0001 complete [ 9674.647174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9687.467329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9697.845890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9711.702416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9737.544384] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 9748.987528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9749.030360] Lustre: lustre-MDT0000: reset Object Index mappings [ 9749.625891] LustreError: 260339:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9749.647629] LustreError: 260339:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9749.734501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9754.657252] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 9755.106116] LustreError: 260340:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9759.207253] LustreError: 260339:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9764.260559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9764.286393] Lustre: lustre-MDT0001: reset Object Index mappings [ 9764.325990] LustreError: 260340:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9764.759166] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9770.964222] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 9773.795963] Lustre: 261484:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9781.282152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9781.910574] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9788.352540] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 9790.134028] LustreError: 261839:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9794.251607] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9797.658167] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9803.259933] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9804.030153] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 9812.040244] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9815.418221] Lustre: 263354:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9821.165228] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9821.175152] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9821.190539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9823.010616] Lustre: server umount lustre-MDT0000 complete [ 9823.718833] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9823.727526] LustreError: 260339:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9823.736073] LustreError: 260339:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9827.137074] LustreError: 260319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789273787 with bad export cookie 1704995937159620449 [ 9827.151453] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9827.153806] LustreError: 260319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9827.618254] Lustre: server umount lustre-MDT0001 complete [ 9842.638346] Lustre: server umount lustre-OST0000 complete [ 9844.703126] Lustre: 250309:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789273789/real 1789273789] req@ffff893b45ce7800 x1876189149003392/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789273805 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9844.738266] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9844.744964] Lustre: Skipped 2 previous similar messages [ 9846.907427] Lustre: server umount lustre-OST0001 complete [ 9873.653276] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 00:30:31 (1789273831) [ 9892.723159] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [ 9903.446816] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [ 9943.516255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9943.823393] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9943.875260] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9943.945859] Lustre: lustre-MDT0000: new disk, initializing [ 9944.019525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9944.023507] Lustre: Skipped 1 previous similar message [ 9944.038724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9949.099105] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 9959.323594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9959.453267] Lustre: 267612:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9959.515948] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9959.522854] Lustre: Skipped 1 previous similar message [ 9959.627704] Lustre: lustre-MDT0001: new disk, initializing [ 9959.806711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9959.830060] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9964.112663] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [ 9973.646516] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9978.927318] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9986.063253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9986.403655] Lustre: lustre-OST0000: new disk, initializing [ 9986.405608] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9986.418700] Lustre: 269576:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9987.787852] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9987.804449] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9987.970237] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9995.001292] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [10009.760415] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10014.952938] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10015.210784] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10020.108774] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10020.331997] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10023.911200] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10023.911785] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10023.927583] LustreError: Skipped 1 previous similar message [10023.928389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10023.946201] Lustre: Skipped 1 previous similar message [10023.962038] Lustre: Skipped 1 previous similar message [10028.532724] Lustre: server umount lustre-OST0000 complete [10034.153364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10034.168224] Lustre: Skipped 1 previous similar message [10038.877283] Lustre: server umount lustre-MDT0000 complete [10039.269338] LustreError: 267623:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10039.289396] LustreError: 267623:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10042.379610] LustreError: 267604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789274003 with bad export cookie 1704995937159622542 [10042.385613] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10042.388535] LustreError: 267604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10042.717426] Lustre: server umount lustre-MDT0001 complete [10061.787429] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [10074.958304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10075.517641] LustreError: 272513:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10075.614767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10075.619331] Lustre: Skipped 2 previous similar messages [10080.742171] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [10089.711303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10096.282613] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [10099.976482] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10109.295525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10120.465415] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [10133.030962] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10133.330892] Lustre: lustre-OST0001: new disk, initializing [10133.336427] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10133.357511] Lustre: 274825:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10134.628144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10134.651485] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10134.881634] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10141.340270] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [10152.193720] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10156.359185] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10163.340746] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10163.365584] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10172.899472] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10172.909231] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10172.926179] Lustre: Skipped 2 previous similar messages [10172.937827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10178.594110] Lustre: server umount lustre-MDT0000 complete [10181.112018] LustreError: 272513:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10181.134611] LustreError: 272513:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10183.318314] LustreError: 272493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789274144 with bad export cookie 1704995937159623424 [10183.341434] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10183.348445] LustreError: 272493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10183.816736] Lustre: server umount lustre-MDT0001 complete [10199.569598] Lustre: server umount lustre-OST0000 complete [10203.103624] Lustre: 250307:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789274147/real 1789274147] req@ffff893a4964e680 x1876189149114752/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789274163 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10204.993637] Lustre: server umount lustre-OST0001 complete [10230.045684] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [10245.835389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10246.732839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10246.749308] Lustre: Skipped 3 previous similar messages [10251.979342] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [10262.230164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10266.788919] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [10269.790281] Lustre: 279038:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10276.848749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10284.045307] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [10291.506451] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10293.641400] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10299.430949] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10302.323601] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [10313.019246] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10317.228110] Lustre: 280916:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10323.707549] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10323.730180] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10335.012112] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 00:38:13 (1789274293) [10354.657590] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10354.667571] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10354.692024] Lustre: Skipped 4 previous similar messages [10354.697860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10354.707280] Lustre: Skipped 4 previous similar messages [10358.789064] Lustre: server umount lustre-MDT0000 complete [10360.809063] LustreError: 277898:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10360.916916] LustreError: 277898:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [10364.339600] LustreError: 277877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789274325 with bad export cookie 1704995937159629759 [10364.359094] LustreError: 277877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10364.372634] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10370.941863] Lustre: server umount lustre-MDT0001 complete [10382.326798] Lustre: server umount lustre-OST0000 complete [10386.149619] Lustre: server umount lustre-OST0001 complete [10391.822576] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [10401.136672] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [10462.387189] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [10476.046904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10476.461450] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10476.550362] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10476.640493] Lustre: lustre-MDT0000: new disk, initializing [10476.784551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10481.825732] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [10492.775787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10492.937080] Lustre: 285163:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10493.210061] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10497.544385] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [10509.197976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10509.429796] Lustre: 286795:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10509.508771] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10509.514342] Lustre: Skipped 5 previous similar messages [10511.135335] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10515.278526] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [10525.377743] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10525.517309] Lustre: 287664:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10527.068217] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10527.079604] Lustre: Skipped 1 previous similar message [10527.185368] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10531.504557] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [10543.182595] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10548.286538] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10548.291553] Lustre: Skipped 1 previous similar message [10565.530133] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 00:42:04 (1789274524) [10568.160214] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10568.183286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10568.190532] Lustre: Skipped 8 previous similar messages [10568.204507] Lustre: Skipped 11 previous similar messages [10570.034813] Lustre: server umount lustre-MDT0000 complete [10573.493442] LustreError: 285156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789274534 with bad export cookie 1704995937159637627 [10573.503866] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10573.513345] LustreError: 285156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10573.808467] Lustre: server umount lustre-MDT0001 complete [10587.681577] Lustre: server umount lustre-OST0000 complete [10600.629562] Lustre: server umount lustre-OST0001 complete [10608.876239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10613.692490] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [10619.876866] LustreError: 289913:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10619.892863] LustreError: 289913:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [10623.204305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10628.912792] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [10638.345223] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10648.725353] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10658.398801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10665.879357] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [10677.477665] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10702.372648] loop1: detected capacity change from 0 to 204800000 [10702.644931] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10702.656318] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10702.675198] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10707.795519] loop1: detected capacity change from 0 to 204800000 [10713.547249] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10713.661250] LustreError: 289901:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs17031: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10713.674265] LustreError: 289901:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs17031 [10713.679685] LustreError: 289901:0:(mgs_handler.c:517:mgs_target_reg()) fs17031-MDT0001: Error checking target index: rc = -13 [10713.687056] LustreError: fs17031-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10713.695138] LustreError: 293632:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [10713.702269] LustreError: 293632:0:(tgt_mount.c:2132:server_put_super()) no obd fs17031-MDT0001 [10713.709077] LustreError: 293632:0:(tgt_mount.c:116:server_deregister_mount()) fs17031-MDT0001 not registered [10713.756700] Lustre: server umount fs17031-MDT0001 complete [10713.761452] LustreError: 293632:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10723.760588] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10723.930177] Lustre: Setting parameter fs17031-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs17031-MDT0001 [10723.958729] Lustre: srv-fs17031-MDT0001: No data found on store. Initialize space. [10723.964106] Lustre: Skipped 4 previous similar messages [10724.151059] Lustre: fs17031-MDT0001: new disk, initializing [10724.162144] Lustre: Skipped 3 previous similar messages [10724.321085] Lustre: srv-fs17031-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10724.333025] Lustre: 294098:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs17031-MDT0001-mdtlov: can't create new llog: rc = -115 [10724.357369] LustreError: 294098:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs17031-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10730.790664] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [10738.888715] Lustre: server umount fs17031-MDT0001 complete [10750.952546] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10755.081236] loop1: detected capacity change from 0 to 204800000 [10761.426835] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10762.043080] Lustre: 295668:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs17031-MDT0001-mdtlov: can't create new llog: rc = -115 [10762.065132] LustreError: 295668:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs17031-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10768.798635] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [10776.437571] Lustre: server umount fs17031-MDT0001 complete [10790.370288] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.14@tcp (stopping) [10791.910717] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10791.924054] Lustre: Skipped 2 previous similar messages [10795.940766] Lustre: server umount lustre-MDT0000 complete [10805.174960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10805.311198] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10810.235825] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [10810.875587] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10818.283282] loop1: detected capacity change from 0 to 204800000 [10824.281103] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10824.743948] Lustre: 298118:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs17031-MDT0001-mdtlov: can't create new llog: rc = -115 [10824.763414] LustreError: 298118:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs17031-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10829.991968] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [10834.506953] Lustre: server umount fs17031-MDT0001 complete [10862.076028] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10862.092260] LustreError: Skipped 1 previous similar message [10874.335171] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10874.531572] Lustre: server umount lustre-OST0000 complete [10884.697586] Lustre: server umount lustre-MDT0000 complete [10888.882386] LustreError: 289894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789274849 with bad export cookie 1704995937159640308 [10888.883892] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10888.891187] LustreError: 289894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10889.603987] Lustre: server umount lustre-MDT0001 complete [10904.352989] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [10908.758805] Key type lgssc unregistered [10909.267839] LNet: 300464:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10909.283056] LNetError: 300464:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10909.306386] LNet: Removed LNI 192.168.203.114@tcp [10910.542575] Key type .llcrypt unregistered [10910.544439] Key type ._llcrypt unregistered [10930.828710] Key type ._llcrypt registered [10930.831726] Key type .llcrypt registered [10930.989932] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [10931.917771] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10932.002671] alg: No test for adler32 (adler32-zlib) [10933.021645] Lustre: Lustre: Build Version: 2.17.57_104_g543d5f9 [10933.319403] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [10935.063254] Key type lgssc registered [10936.094733] Lustre: Echo OBD driver; http://www.lustre.org/ [10944.799635] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10944.841869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10946.590160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10952.403183] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [10956.773355] LustreError: 301735:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10956.805464] LustreError: 301735:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10961.888920] LustreError: 301734:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10962.351197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10962.847187] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10967.822703] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [10975.617957] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10982.331755] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10989.802338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10990.193698] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10992.244279] LustreError: 303565:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10995.852463] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [10997.739795] LustreError: 303563:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11002.852524] LustreError: 303564:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11005.253117] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11012.566972] LustreError: 303563:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11012.611674] LustreError: 303563:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11026.996703] loop1: detected capacity change from 0 to 204800000 [11027.224196] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11027.238225] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11027.248938] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11029.492974] LustreError: 303564:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11029.526577] LustreError: 303564:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [11033.308213] loop1: detected capacity change from 0 to 204800000 [11039.512338] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11039.716518] Lustre: fs9437-OST0000: new disk, initializing [11039.725272] Lustre: srv-fs9437-OST0000: No data found on store. Initialize space. [11039.735728] Lustre: 305474:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11039.796462] Lustre: fs9437-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11046.411213] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [11054.470640] Lustre: server umount fs9437-OST0000 complete [11063.752628] LustreError: 303563:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11063.777305] LustreError: 303563:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 18 previous similar messages [11066.708325] loop1: detected capacity change from 0 to 204800000 [11070.438304] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11072.821096] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11073.203285] Lustre: fs9437-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11082.519731] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [11090.939219] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11095.861560] loop2: detected capacity change from 0 to 204800000 [11096.174642] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11096.206214] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11096.232521] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11102.245232] loop2: detected capacity change from 0 to 204800000 [11109.049370] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11109.170579] LustreError: fs9437-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11109.178225] LustreError: fs9437-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11109.183862] LustreError: 308513:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [11109.189462] LustreError: 308513:0:(tgt_mount.c:2132:server_put_super()) no obd fs9437-OST0001 [11109.193301] LustreError: 308513:0:(tgt_mount.c:116:server_deregister_mount()) fs9437-OST0001 not registered [11109.211297] Lustre: server umount fs9437-OST0001 complete [11109.214702] LustreError: 308513:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11120.797931] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11120.990437] Lustre: fs9437-OST0001: new disk, initializing [11120.996331] Lustre: srv-fs9437-OST0001: No data found on store. Initialize space. [11121.016411] Lustre: 309064:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11121.141235] Lustre: fs9437-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11129.590622] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [11130.311620] LustreError: 303707:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11130.332427] LustreError: 303707:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [11139.509233] Lustre: server umount fs9437-OST0000 complete [11149.406206] Lustre: server umount fs9437-OST0001 complete [11163.472507] loop1: detected capacity change from 0 to 204800000 [11169.594438] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11169.921176] Lustre: fs9437-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11176.388920] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [11182.711696] loop2: detected capacity change from 0 to 204800000 [11188.279978] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11195.228814] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [11204.857982] Lustre: server umount fs9437-OST0000 complete [11216.301785] Lustre: server umount fs9437-OST0001 complete [11224.054225] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11233.252406] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11233.273875] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11233.298965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11234.272865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11234.278463] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11239.417399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11239.424869] Lustre: Skipped 2 previous similar messages [11244.000570] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11244.218421] Lustre: server umount lustre-OST0000 complete [11249.635411] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11249.641613] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11249.654394] LustreError: Skipped 1 previous similar message [11249.667102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11249.675606] Lustre: Skipped 1 previous similar message [11255.048425] Lustre: server umount lustre-MDT0000 complete [11259.207486] LustreError: 301715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789275220 with bad export cookie 7036385025613019041 [11259.212942] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11259.233463] LustreError: 301715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11259.736455] Lustre: server umount lustre-MDT0001 complete [11273.441429] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [11277.646995] Key type lgssc unregistered [11278.184706] LNet: 314986:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11278.193830] LNetError: 314986:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11278.219810] LNet: Removed LNI 192.168.203.114@tcp [11279.592176] Key type .llcrypt unregistered [11279.597896] Key type ._llcrypt unregistered [11314.750679] Key type ._llcrypt registered [11314.753707] Key type .llcrypt registered [11314.848552] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [11332.327238] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [11333.990166] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11334.040103] alg: No test for adler32 (adler32-zlib) [11335.437906] Lustre: Lustre: Build Version: 2.17.57_104_g543d5f9 [11335.896549] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [11337.631295] Key type lgssc registered [11338.618815] Lustre: Echo OBD driver; http://www.lustre.org/ [11376.914359] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11376.956479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11378.276880] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11378.320191] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11378.384875] Lustre: lustre-MDT0000: new disk, initializing [11378.468811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11378.489502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11383.054993] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [11392.740255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11392.884940] Lustre: 318842:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11392.927200] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11392.930803] Lustre: Skipped 1 previous similar message [11393.013862] Lustre: lustre-MDT0001: new disk, initializing [11393.134795] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11393.189415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11393.196544] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11396.983425] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [11406.504120] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11412.359268] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11419.459796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11419.742693] Lustre: lustre-OST0000: new disk, initializing [11419.747039] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11419.752105] Lustre: 320811:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11419.836255] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11425.872462] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [11429.424532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11429.431137] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11429.566962] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11435.117966] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11437.634484] Lustre: Mounted lustre-client - version 2.17.57_104_g543d5f9 [11438.634592] Lustre: Unmounted lustre-client [11447.791351] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11447.796685] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11447.810756] Lustre: Skipped 1 previous similar message [11447.827045] Lustre: Skipped 1 previous similar message [11452.907115] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11452.920063] Lustre: Skipped 1 previous similar message [11453.912728] Lustre: server umount lustre-OST0000 complete [11458.019863] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11458.031149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11458.063053] Lustre: Skipped 1 previous similar message [11458.069493] Lustre: Skipped 1 previous similar message [11463.137925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11463.143127] Lustre: Skipped 1 previous similar message [11468.256892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11468.269034] Lustre: Skipped 1 previous similar message [11472.351195] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11472.648157] Lustre: server umount lustre-MDT0000 complete [11476.672100] LustreError: 318835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789275437 with bad export cookie 3750880223629640337 [11476.674405] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11476.680407] LustreError: 318835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11477.091545] Lustre: server umount lustre-MDT0001 complete [11487.975536] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [11491.083916] Key type lgssc unregistered [11492.450159] LNet: 322972:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11492.460457] LNetError: 322972:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11568.236512] LNet: Removed LNI 192.168.203.114@tcp [11569.337841] Key type .llcrypt unregistered [11569.349843] Key type ._llcrypt unregistered [11587.474930] Key type ._llcrypt registered [11587.477871] Key type .llcrypt registered [11587.685697] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [11602.792798] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [11604.357898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11604.464470] alg: No test for adler32 (adler32-zlib) [11605.539544] Lustre: Lustre: Build Version: 2.17.57_104_g543d5f9 [11605.790026] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [11607.464254] Key type lgssc registered [11608.607971] Lustre: Echo OBD driver; http://www.lustre.org/ [11640.559127] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 00:59:59 (1789275599) [11647.995447] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11648.036208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11649.313517] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11649.339791] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11649.434585] Lustre: lustre-MDT0000: new disk, initializing [11649.509763] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11649.530863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11653.609226] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [11663.057028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11663.173849] Lustre: 326190:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11663.230259] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11663.234803] Lustre: Skipped 1 previous similar message [11663.313863] Lustre: lustre-MDT0001: new disk, initializing [11663.405082] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11663.455247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11663.463442] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11667.705216] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [11676.733170] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11683.260911] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11692.070445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11692.404823] Lustre: lustre-OST0000: new disk, initializing [11692.407717] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11692.414538] Lustre: 328156:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11692.501325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11698.845598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11698.862587] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11699.023401] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11700.781421] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [11711.713496] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11729.905632] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11729.918627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11729.927790] Lustre: Skipped 1 previous similar message [11729.933226] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11729.935175] Lustre: Skipped 1 previous similar message [11733.107175] Lustre: server umount lustre-OST0000 complete [11740.128240] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11740.131635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11740.163725] Lustre: Skipped 1 previous similar message [11743.600902] Lustre: server umount lustre-MDT0000 complete [11748.235753] LustreError: 326181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789275709 with bad export cookie 15686224957274896473 [11748.239482] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11748.248288] LustreError: 326181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11748.657176] Lustre: server umount lustre-MDT0001 complete [11761.252261] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [11763.471547] Key type lgssc unregistered [11763.729588] LNet: 330080:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11763.734670] LNetError: 330080:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11763.752856] LNet: Removed LNI 192.168.203.114@tcp [11764.560302] Key type .llcrypt unregistered [11764.562692] Key type ._llcrypt unregistered [11785.609978] Key type ._llcrypt registered [11785.611784] Key type .llcrypt registered [11785.741510] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [11786.802849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11786.823625] alg: No test for adler32 (adler32-zlib) [11787.936532] Lustre: Lustre: Build Version: 2.17.57_104_g543d5f9 [11788.231132] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [11789.951789] Key type lgssc registered [11791.237409] Lustre: Echo OBD driver; http://www.lustre.org/ [11799.663471] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11799.691071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11801.321347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11806.137667] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [11811.808471] LustreError: 331349:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11811.836787] LustreError: 331349:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11815.986064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11816.479878] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11821.792051] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [11830.306751] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11836.948951] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11844.455979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11844.835078] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11850.933869] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [11858.655637] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11870.099189] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11872.637657] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11881.940441] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.14@tcp (stopping) [11881.953925] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11881.960223] Lustre: Skipped 2 previous similar messages [11884.656317] Lustre: server umount lustre-MDT0000 complete [11887.046804] LustreError: 331345:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11889.871292] LustreError: 331331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789275850 with bad export cookie 5755065428675496050 [11889.880985] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11889.896121] LustreError: 331331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11892.165706] LustreError: 331346:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11892.206052] LustreError: 331346:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11892.211902] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.14@tcp (stopping) [11892.216405] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11892.222314] Lustre: Skipped 3 previous similar messages [11894.240762] LustreError: 333606:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11894.241306] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11894.275786] Lustre: Skipped 1 previous similar message [11896.301624] Lustre: server umount lustre-MDT0001 complete [11900.617252] Lustre: server umount lustre-OST0000 complete [11909.280727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11914.011514] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [11917.365338] Lustre: 335535:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.114@tcp [11919.796163] Lustre: server umount MGS complete [11928.415962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11929.013985] LustreError: 336108:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11929.140853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11934.000560] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [11937.223602] LustreError: 336103:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.14@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11937.243861] LustreError: 336103:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11943.088345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11943.402490] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11948.902384] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [11958.444588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11958.721284] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11965.866233] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [11974.534909] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11976.368276] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11982.881080] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11984.985384] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11991.015505] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [11991.024552] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11991.044110] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11992.499356] Lustre: server umount lustre-OST0000 complete [11999.715544] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11999.719429] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11999.725491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12002.727236] Lustre: server umount lustre-MDT0000 complete [12004.848329] LustreError: 337060:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12004.880425] LustreError: 337060:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12006.948472] LustreError: 336090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789275967 with bad export cookie 5755065428675496890 [12006.957285] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12006.967115] LustreError: 336090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12016.855274] Lustre: server umount lustre-MDT0001 complete [12027.970814] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [12031.024996] Key type lgssc unregistered [12031.312088] LNet: 339546:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12031.319460] LNetError: 339546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12031.351589] LNet: Removed LNI 192.168.203.114@tcp [12032.126944] Key type .llcrypt unregistered [12032.129509] Key type ._llcrypt unregistered [12051.110738] Key type ._llcrypt registered [12051.114783] Key type .llcrypt registered [12051.386738] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [12068.071470] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [12069.064267] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12069.105092] alg: No test for adler32 (adler32-zlib) [12070.157272] Lustre: Lustre: Build Version: 2.17.57_104_g543d5f9 [12070.353945] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [12072.047295] Key type lgssc registered [12073.306626] Lustre: Echo OBD driver; http://www.lustre.org/ [12111.471128] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12111.507805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12112.700607] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12112.728624] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12112.814382] Lustre: lustre-MDT0000: new disk, initializing [12112.907162] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12112.923172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12117.093403] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [12126.292848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12126.394492] Lustre: 342706:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12126.443276] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12126.446580] Lustre: Skipped 1 previous similar message [12126.500224] Lustre: lustre-MDT0001: new disk, initializing [12126.571946] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12126.607962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12126.629622] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12130.884511] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [12140.774977] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12146.061090] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12153.193260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12153.484120] Lustre: lustre-OST0000: new disk, initializing [12153.495494] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12153.503871] Lustre: 344672:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12153.646635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12161.124108] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12161.141256] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12161.303637] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12161.526357] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [12173.947348] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12180.884889] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12181.318780] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12187.565130] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12187.965504] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12192.233867] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12192.240145] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12192.261508] Lustre: Skipped 1 previous similar message [12192.270256] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12192.271829] Lustre: Skipped 1 previous similar message [12196.413462] Lustre: server umount lustre-OST0000 complete [12202.466470] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12202.471750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12202.478145] Lustre: Skipped 1 previous similar message [12202.492018] Lustre: Skipped 1 previous similar message [12206.819577] Lustre: server umount lustre-MDT0000 complete [12210.782305] LustreError: 342698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789276171 with bad export cookie 17343063901868677440 [12210.787572] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12210.800466] LustreError: 342698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12211.139890] Lustre: server umount lustre-MDT0001 complete [12219.327935] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 01:09:38 (1789276178) [12235.378051] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [12246.631910] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [12284.219309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12284.444109] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12284.486445] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12284.568129] Lustre: lustre-MDT0000: new disk, initializing [12284.664899] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12284.682788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12289.604480] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [12300.707045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12300.834468] Lustre: 349433:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12300.926246] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12300.937882] Lustre: Skipped 1 previous similar message [12301.063847] Lustre: lustre-MDT0001: new disk, initializing [12301.197584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12301.249272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12301.258601] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12305.620824] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [12315.184629] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12322.073473] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12330.361617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12330.712578] Lustre: lustre-OST0000: new disk, initializing [12330.724543] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12330.739085] Lustre: 351403:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12330.899776] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12338.404455] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [12339.795300] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12339.816921] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12340.034393] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12349.851853] Lustre: DEBUG MARKER: oleg314-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12386.297258] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12386.310280] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12386.319407] LustreError: Skipped 2 previous similar messages [12386.320547] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12386.362287] Lustre: Skipped 1 previous similar message [12388.121709] Lustre: server umount lustre-OST0000 complete [12393.443754] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12393.453698] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12393.467616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12393.474811] Lustre: Skipped 1 previous similar message [12398.965718] Lustre: server umount lustre-MDT0000 complete [12401.635246] LustreError: 349445:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12401.664229] LustreError: 349445:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12402.998146] LustreError: 351401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789276363 with bad export cookie 17343063901868678322 [12403.010279] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12403.014074] LustreError: 351401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12403.356976] Lustre: server umount lustre-MDT0001 complete [12414.678607] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [12418.058954] Key type lgssc unregistered [12418.431059] LNet: 353554:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12418.436535] LNetError: 353554:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12418.456067] LNet: Removed LNI 192.168.203.114@tcp [12419.358694] Key type .llcrypt unregistered [12419.362366] Key type ._llcrypt unregistered [12516.774303] Key type ._llcrypt registered [12516.776439] Key type .llcrypt registered [12516.856632] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [12517.116324] Key type .llcrypt unregistered [12517.122443] Key type ._llcrypt unregistered [12597.836391] Key type ._llcrypt registered [12597.838758] Key type .llcrypt registered [12597.926928] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [12598.289435] Key type .llcrypt unregistered [12598.291351] Key type ._llcrypt unregistered [12630.449727] Key type ._llcrypt registered [12630.452627] Key type .llcrypt registered [12630.573088] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [12630.857191] Key type .llcrypt unregistered [12630.859648] Key type ._llcrypt unregistered [12662.821513] Key type ._llcrypt registered [12662.825603] Key type .llcrypt registered [12662.934527] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [12663.323559] Key type .llcrypt unregistered [12663.328947] Key type ._llcrypt unregistered [12743.240279] Key type ._llcrypt registered [12743.243123] Key type .llcrypt registered [12743.340803] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [12743.620505] Key type .llcrypt unregistered [12743.624631] Key type ._llcrypt unregistered [12773.705857] Key type ._llcrypt registered [12773.709955] Key type .llcrypt registered [12773.815277] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [12774.204444] Key type .llcrypt unregistered [12774.213864] Key type ._llcrypt unregistered [12805.190192] Key type ._llcrypt registered [12805.193454] Key type .llcrypt registered [12805.301886] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [12805.599320] Key type .llcrypt unregistered [12805.603919] Key type ._llcrypt unregistered [12836.121939] Key type ._llcrypt registered [12836.123877] Key type .llcrypt registered [12836.306136] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [12836.627409] Key type .llcrypt unregistered [12836.630585] Key type ._llcrypt unregistered [12866.041125] Key type ._llcrypt registered [12866.044452] Key type .llcrypt registered [12866.131887] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [12866.368523] Key type .llcrypt unregistered [12866.373218] Key type ._llcrypt unregistered [12895.621132] Key type ._llcrypt registered [12895.625123] Key type .llcrypt registered [12895.721220] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [12896.086669] Key type .llcrypt unregistered [12896.090455] Key type ._llcrypt unregistered [12958.168472] Key type ._llcrypt registered [12958.172991] Key type .llcrypt registered [12958.314399] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [12958.772293] Key type .llcrypt unregistered [12958.773934] Key type ._llcrypt unregistered [12990.990890] Key type ._llcrypt registered [12990.993498] Key type .llcrypt registered [12991.100970] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [12991.418480] Key type .llcrypt unregistered [12991.426061] Key type ._llcrypt unregistered [13021.106334] Key type ._llcrypt registered [13021.109764] Key type .llcrypt registered [13021.275846] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [13021.513337] Key type .llcrypt unregistered [13021.517885] Key type ._llcrypt unregistered [13078.467285] Key type ._llcrypt registered [13078.472482] Key type .llcrypt registered [13078.635670] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [13096.634791] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [13097.794795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13097.841396] alg: No test for adler32 (adler32-zlib) [13098.824649] Lustre: Lustre: Build Version: 2.17.57_104_g543d5f9 [13099.030839] LNet: Added LNI 192.168.203.114@tcp [8/256/0/180] [13100.719182] Key type lgssc registered [13102.132544] Lustre: Echo OBD driver; http://www.lustre.org/ [13135.543881] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13135.584541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13136.834779] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13136.878326] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13136.959693] Lustre: tst250fs-MDT0000: new disk, initializing [13137.037874] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13137.053830] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13141.949887] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [13152.805622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13152.954869] Lustre: 373848:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13153.042584] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13153.050480] Lustre: Skipped 1 previous similar message [13153.198792] Lustre: tst250fs-MDT0001: new disk, initializing [13153.289499] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13153.325728] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13153.332529] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13157.836680] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [13162.782881] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13170.011830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13170.274652] Lustre: tst250fs-OST0000: new disk, initializing [13170.280497] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13170.287293] Lustre: 375485:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13170.371410] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13170.614277] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13170.627887] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13170.805750] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13177.020189] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [13187.811847] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13188.022917] Lustre: tst250fs-OST0001: new disk, initializing [13188.031796] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13188.042526] Lustre: 376356:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13188.178967] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13193.875476] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all [13197.974882] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13197.993936] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13198.130374] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13199.362653] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13201.806468] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [13201.822295] Lustre: Skipped 2 previous similar messages [13223.905825] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13223.914984] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13223.932303] Lustre: Skipped 3 previous similar messages [13223.940578] Lustre: Skipped 2 previous similar messages [13227.409718] Lustre: server umount tst250fs-MDT0000 complete [13230.986715] LustreError: 373843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789277191 with bad export cookie 4207195283424961850 [13230.989320] LustreError: MGC192.168.203.114@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13231.023647] LustreError: 373843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13231.336730] Lustre: server umount tst250fs-MDT0001 complete [13245.784489] Lustre: server umount tst250fs-OST0000 complete [13258.506632] Lustre: server umount tst250fs-OST0001 complete [13277.257347] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [13286.015919] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [13324.751154] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 01:28:03 (1789277283) [13326.577384] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13344.840479] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_hostid [13355.216852] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing load_modules_local [13390.031838] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13219 sec ========= 01:29:08 (1789277348) [13392.012251] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 01:29:10 (1789277350) === [13395.898762] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 01:29:14 (1789277354) === [13437.287706] Lustre: DEBUG MARKER: oleg314-server.virtnet: executing unload_modules_local [13441.050323] Key type lgssc unregistered [13441.426492] LNet: 385484:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13441.440816] LNetError: 385484:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13441.466478] LNet: Removed LNI 192.168.203.114@tcp [13442.444158] Key type .llcrypt unregistered [13442.450533] Key type ._llcrypt unregistered