[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 852958270 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.988 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003115] x2apic enabled [ 0.004006] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.008515] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22982c12b6d, max_idle_ns: 440795281273 ns [ 0.009021] Calibrating delay loop (skipped) preset value.. 4799.97 BogoMIPS (lpj=2399988) [ 0.010015] pid_max: default: 32768 minimum: 301 [ 0.011125] LSM: Security Framework initializing [ 0.013054] Yama: becoming mindful. [ 0.014036] SELinux: Initializing. [ 0.015064] *** VALIDATE selinux *** [ 0.023797] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028464] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029154] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030100] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032105] *** VALIDATE tmpfs *** [ 0.033431] *** VALIDATE proc *** [ 0.035134] *** VALIDATE cgroup *** [ 0.036010] *** VALIDATE cgroup2 *** [ 0.037253] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038148] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040031] Spectre V2 : User space: Vulnerable [ 0.041008] Speculative Store Bypass: Vulnerable [ 0.043575] debug: unmapping init [mem 0xffffffff91459000-0xffffffff91460fff] [ 0.045262] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046713] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047025] ... version: 2 [ 0.048011] ... bit width: 48 [ 0.049012] ... generic registers: 4 [ 0.050012] ... value mask: 0000ffffffffffff [ 0.051014] ... max period: 00007fffffffffff [ 0.052014] ... fixed-purpose events: 3 [ 0.053013] ... event mask: 000000070000000f [ 0.054402] rcu: Hierarchical SRCU implementation. [ 0.056606] smp: Bringing up secondary CPUs ... [ 0.057589] x86: Booting SMP configuration: [ 0.058024] .... node #0, CPUs: #1 #2 #3 [ 0.071732] smp: Brought up 1 node, 4 CPUs [ 0.073016] smpboot: Max logical packages: 1 [ 0.074019] smpboot: Total of 4 processors activated (19199.90 BogoMIPS) [ 0.121256] node 0 deferred pages initialised in 45ms [ 0.126186] devtmpfs: initialized [ 0.127285] x86/mm: Memory block size: 128MB [ 0.131513] gcov: version magic: 0x41383552 [ 0.133417] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.134086] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.136373] pinctrl core: initialized pinctrl subsystem [ 0.137209] [ 0.138014] ************************************************************* [ 0.139014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.140013] ** ** [ 0.141041] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.142018] ** ** [ 0.143016] ** This means that this kernel is built to expose internal ** [ 0.144014] ** IOMMU data structures, which may compromise security on ** [ 0.145015] ** your system. ** [ 0.146014] ** ** [ 0.147018] ** If you see this message and you are not debugging the ** [ 0.148017] ** kernel, report this immediately to your vendor! ** [ 0.149018] ** ** [ 0.150018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.151015] ************************************************************* [ 0.153940] NET: Registered protocol family 16 [ 0.156462] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.157141] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.158075] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.160103] cpuidle: using governor menu [ 0.165000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.174042] PCI: Using configuration type 1 for base access [ 0.180143] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.194451] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.196020] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.207032] cryptd: max_cpu_qlen set to 1000 [ 0.212250] ACPI: Added _OSI(Module Device) [ 0.217017] ACPI: Added _OSI(Processor Device) [ 0.221018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.227016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.239558] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.265174] ACPI: Interpreter enabled [ 0.266062] ACPI: PM: (supports S0 S3 S4 S5) [ 0.267017] ACPI: Using IOAPIC for interrupt routing [ 0.268129] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.269888] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.293145] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.294047] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.295017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.296087] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.298516] acpiphp: Slot [2] registered [ 0.299174] acpiphp: Slot [5] registered [ 0.300226] acpiphp: Slot [6] registered [ 0.302054] acpiphp: Slot [7] registered [ 0.303164] acpiphp: Slot [8] registered [ 0.304130] acpiphp: Slot [9] registered [ 0.305103] acpiphp: Slot [10] registered [ 0.306144] acpiphp: Slot [3] registered [ 0.307146] acpiphp: Slot [4] registered [ 0.308126] acpiphp: Slot [11] registered [ 0.310059] acpiphp: Slot [12] registered [ 0.311162] acpiphp: Slot [13] registered [ 0.312857] acpiphp: Slot [14] registered [ 0.313139] acpiphp: Slot [15] registered [ 0.314143] acpiphp: Slot [16] registered [ 0.315500] acpiphp: Slot [17] registered [ 0.316170] acpiphp: Slot [18] registered [ 0.317132] acpiphp: Slot [19] registered [ 0.319051] acpiphp: Slot [20] registered [ 0.320136] acpiphp: Slot [21] registered [ 0.322033] acpiphp: Slot [22] registered [ 0.324097] acpiphp: Slot [23] registered [ 0.325146] acpiphp: Slot [24] registered [ 0.326877] acpiphp: Slot [25] registered [ 0.327136] acpiphp: Slot [26] registered [ 0.328134] acpiphp: Slot [27] registered [ 0.329134] acpiphp: Slot [28] registered [ 0.330130] acpiphp: Slot [29] registered [ 0.332099] acpiphp: Slot [30] registered [ 0.333140] acpiphp: Slot [31] registered [ 0.334203] PCI host bridge to bus 0000:00 [ 0.335021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.336027] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.337022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.338027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.339026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.340148] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.341195] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.343522] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.345720] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.352016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.355072] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.356030] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.357022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.358023] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.359569] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.362018] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.363047] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.364859] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.367017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.376015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.379015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.383964] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.386020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.389016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.396017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.403588] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.406016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.409017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.417024] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.431000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.452016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.468000] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.497070] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.518935] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.536016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.552018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.583016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.605942] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.627015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.647014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.672000] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.687060] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.697000] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.710022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.741016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.766469] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.770692] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.774707] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.778451] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.781321] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.788060] iommu: Default domain type: Passthrough [ 0.789684] SCSI subsystem initialized [ 0.792126] ACPI: bus type USB registered [ 0.794222] usbcore: registered new interface driver usbfs [ 0.798168] usbcore: registered new interface driver hub [ 0.800085] usbcore: registered new device driver usb [ 0.801202] pps_core: LinuxPPS API ver. 1 registered [ 0.802000] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.803061] PTP clock support registered [ 0.809961] EDAC MC: Ver: 3.0.0 [ 0.811111] PCI: Using ACPI for IRQ routing [ 0.814098] NetLabel: Initializing [ 0.816008] NetLabel: domain hash size = 128 [ 0.818010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.822084] NetLabel: unlabeled traffic allowed by default [ 0.825039] vgaarb: loaded [ 0.830507] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.833016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.841791] clocksource: Switched to clocksource kvm-clock [ 1.076607] VFS: Disk quotas dquot_6.6.0 [ 1.082707] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.087788] *** VALIDATE ramfs *** [ 1.089720] *** VALIDATE hugetlbfs *** [ 1.092793] pnp: PnP ACPI init [ 1.096422] pnp: PnP ACPI: found 6 devices [ 1.151696] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.158502] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.161976] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.169985] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.180259] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.192988] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.207145] NET: Registered protocol family 2 [ 1.216102] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.232763] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.242190] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.253313] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.261514] TCP: Hash tables configured (established 65536 bind 65536) [ 1.270471] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.281566] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.294986] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.299845] NET: Registered protocol family 1 [ 1.304477] RPC: Registered named UNIX socket transport module. [ 1.314096] RPC: Registered udp transport module. [ 1.317968] RPC: Registered tcp transport module. [ 1.320863] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.329857] NET: Registered protocol family 44 [ 1.336927] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.339161] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.348298] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.353718] PCI: CLS 0 bytes, default 64 [ 1.371402] Unpacking initramfs... [ 6.509272] debug: unmapping init [mem 0xffff8ccb3cc54000-0xffff8ccb3ffbffff] [ 6.529159] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 6.540874] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 6.554941] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22982c12b6d, max_idle_ns: 440795281273 ns [ 7.075250] hrtimer: interrupt took 3240330 ns [ 9.003315] Initialise system trusted keyrings [ 9.007213] Key type blacklist registered [ 9.009629] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 9.032592] zbud: loaded [ 9.038377] *** VALIDATE nfs *** [ 9.040865] *** VALIDATE nfs4 *** [ 9.043140] pstore: using deflate compression [ 9.069668] Platform Keyring initialized [ 9.666611] NET: Registered protocol family 38 [ 9.670127] Key type asymmetric registered [ 9.672457] Asymmetric key parser 'x509' registered [ 9.675502] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 9.687358] io scheduler mq-deadline registered [ 9.690898] io scheduler kyber registered [ 9.694706] io scheduler bfq registered [ 9.699899] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 9.702026] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 9.705372] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 9.708392] ACPI: Power Button [PWRF] [ 9.719777] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 9.731974] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 9.748887] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 9.758867] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 9.794878] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 9.828080] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 9.878153] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 9.883249] Non-volatile memory driver v1.3 [ 9.884970] Linux agpgart interface v0.103 [ 9.927284] virtio_blk virtio1: [vda] 134768 512-byte logical blocks (69.0 MB/65.8 MiB) [ 9.930251] vda: detected capacity change from 0 to 69001216 [ 9.947449] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 9.950669] vdb: detected capacity change from 0 to 1073741824 [ 9.967373] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 9.971507] vdc: detected capacity change from 0 to 2621440000 [ 9.995212] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 9.997800] vdd: detected capacity change from 0 to 2621440000 [ 10.046910] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 10.054245] vde: detected capacity change from 0 to 4294967296 [ 10.089381] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 10.095110] vdf: detected capacity change from 0 to 4294967296 [ 10.112893] libphy: Fixed MDIO Bus: probed [ 10.150780] usbcore: registered new interface driver usbserial_generic [ 10.155120] usbserial: USB Serial support registered for generic [ 10.157393] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 10.172410] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 10.176761] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 10.187226] mousedev: PS/2 mouse device common for all mice [ 10.198766] rtc_cmos 00:05: RTC can wake from S4 [ 10.210258] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 10.219882] rtc_cmos 00:05: registered as rtc0 [ 10.221281] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 10.226734] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 10.231851] intel_pstate: CPU model not supported [ 10.241249] hid: raw HID events driver (C) Jiri Kosina [ 10.244342] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 10.265310] usbcore: registered new interface driver usbhid [ 10.267152] usbhid: USB HID core driver [ 10.268577] drop_monitor: Initializing network drop monitor service [ 10.290550] Initializing XFRM netlink socket [ 10.297219] NET: Registered protocol family 10 [ 10.307126] Segment Routing with IPv6 [ 10.308501] NET: Registered protocol family 17 [ 10.319441] mpls_gso: MPLS GSO support [ 10.343640] RAS: Correctable Errors collector initialized. [ 10.345641] AVX version of gcm_enc/dec engaged. [ 10.356627] AES CTR mode by8 optimization enabled [ 10.668771] sched_clock: Marking stable (10668691251, 0)->(12530274921, -1861583670) [ 10.684238] registered taskstats version 1 [ 10.690108] Loading compiled-in X.509 certificates [ 10.692834] zswap: loaded using pool lzo/zbud [ 10.772712] Key type big_key registered [ 10.826166] Key type encrypted registered [ 10.827789] ima: No TPM chip found, activating TPM-bypass! [ 10.835425] ima: Allocated hash algorithm: sha1 [ 10.837167] ima: No architecture policies found [ 10.843449] evm: Initialising EVM extended attributes: [ 10.848316] evm: security.selinux [ 10.849470] evm: security.ima [ 10.854136] evm: security.capability [ 10.855790] evm: HMAC attrs: 0x1 [ 10.863922] rtc_cmos 00:05: setting system clock to 2026-05-27 08:13:32 UTC (1779869612) [ 10.893728] debug: unmapping init [mem 0xffffffff92403000-0xffffffff925fffff] [ 10.909529] debug: unmapping init [mem 0xffffffff91182000-0xffffffff91458fff] [ 10.920108] Write protecting the kernel read-only data: 28672k [ 10.926781] debug: unmapping init [mem 0xffffffff8f803000-0xffffffff8f9fffff] [ 10.937706] debug: unmapping init [mem 0xffffffff90114000-0xffffffff901fffff] [ 11.092377] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.126608] systemd[1]: Detected virtualization kvm. [ 11.142472] systemd[1]: Detected architecture x86-64. [ 11.145503] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 11.278157] systemd[1]: No hostname configured. [ 11.285343] systemd[1]: Set hostname to . [ 11.294587] random: systemd: uninitialized urandom read (16 bytes read) [ 11.312175] systemd[1]: Initializing machine ID from random generator. [ 11.528445] random: ln: uninitialized urandom read (6 bytes read) [ 11.807854] random: systemd: uninitialized urandom read (16 bytes read) [ 11.813548] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 11.825406] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 11.835645] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 13.923405] device-mapper: uevent: version 1.0.3 [ 13.933390] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ 16.748778] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 16.917001] random: fast init done [ 17.282304] scsi host0: ata_piix [ 17.304123] scsi host1: ata_piix [ 17.310336] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 17.320820] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 22.429396] random: crng init done [ 22.432760] random: 7 urandom warning(s) missed due to ratelimiting [ 26.753687] dracut-initqueue[594]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 28.743320] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 33.667482] printk: systemd: 26 output lines suppressed due to ratelimiting [ 34.986913] SELinux: Disabled at runtime. [ 35.115587] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 35.146359] systemd[1]: Detected virtualization kvm. [ 35.157053] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 37.717849] systemd[1]: initrd-switch-root.service: Succeeded. [ 37.728140] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 37.741070] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 37.750620] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 37.756715] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 37.784375] systemd[1]: Starting Journal Service... Starting Journal Service... [ 37.809366] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... [ 37.930918] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Paths. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 40.465344] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 42.051491] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 42.176665] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 42.562859] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 42.877118] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (9s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root support (11s / no limit)[ 49.148817] Key type dns_resolver registered [ **] A start job is running for Configur…only root support (11s / no limit)[ 49.661576] NFS: Registering the id_resolver key type [ 49.665980] Key type id_resolver registered [ 49.670163] Key type id_legacy registered [ *] A start job is running for Configur…only root support (12s / no limit) [ **] A start job is running for Configur…only root support (12s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Login Service. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg456-server login: [ 99.936882] libcfs: loading out-of-tree module taints kernel. [ 99.974397] Key type ._llcrypt registered [ 99.978504] Key type .llcrypt registered [ 100.070107] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [ 110.726919] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 111.634296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 111.659631] alg: No test for adler32 (adler32-zlib) [ 112.891481] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 113.590492] LNet: Added LNI 192.168.204.156@tcp [8/256/0/180] [ 115.384155] Key type lgssc registered [ 116.406713] Lustre: Echo OBD driver; http://www.lustre.org/ [ 127.722382] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 155.551504] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 164.147550] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 164.175976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 165.399397] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 165.449362] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 165.513275] Lustre: lustre-MDT0000: new disk, initializing [ 165.586200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 165.601070] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 168.235109] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 177.722977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 177.786956] Lustre: 6481:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 177.812620] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 177.821521] Lustre: Skipped 1 previous similar message [ 177.882547] Lustre: lustre-MDT0001: new disk, initializing [ 177.946169] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 177.964554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 177.970106] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 180.766346] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 184.656762] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 191.599971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 191.754542] Lustre: lustre-OST0000: new disk, initializing [ 191.758301] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 191.763326] Lustre: 8385:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 191.824606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 192.973254] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 192.979123] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 193.055409] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 195.916695] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 207.953391] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 208.056400] Lustre: lustre-OST0001: new disk, initializing [ 208.064557] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 208.069351] Lustre: 9441:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 208.129551] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 212.418196] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 214.029017] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 214.041420] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 214.072136] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 221.386706] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 227.094644] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 231.961831] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing check_logdir /tmp/testlogs/ [ 235.447228] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing yml_node [ 238.556875] Lustre: DEBUG MARKER: Client: 2.17.53.24 [ 240.313404] Lustre: DEBUG MARKER: MDS: 2.17.53.24 [ 242.044891] Lustre: DEBUG MARKER: OSS: 2.17.53.24 [ 243.145951] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed May 27 04:17:24 EDT 2026 [ 256.623926] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 257.777598] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 275.425136] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 275.425785] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 275.433379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 275.439910] Lustre: Skipped 3 previous similar messages [ 275.442912] Lustre: Skipped 2 previous similar messages [ 277.726284] Lustre: server umount lustre-MDT0000 complete [ 280.257437] LustreError: 6474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779869881 with bad export cookie 17663580181235703731 [ 280.259595] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 280.270057] LustreError: 6474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 280.557938] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 280.564334] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 280.570939] Lustre: Skipped 1 previous similar message [ 280.583403] Lustre: Skipped 2 previous similar messages [ 285.669716] LustreError: 9857:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 285.669864] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 285.694261] LustreError: 9857:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 285.710197] Lustre: Skipped 1 previous similar message [ 286.620337] Lustre: server umount lustre-MDT0001 complete [ 290.014062] Lustre: server umount lustre-OST0000 complete [ 293.185881] Lustre: server umount lustre-OST0001 complete [ 298.640351] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [ 306.088430] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 334.713522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 334.916564] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 334.940738] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 335.026479] Lustre: lustre-MDT0000: new disk, initializing [ 335.139945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 335.181286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 338.575513] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 347.564433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 347.667809] Lustre: 14949:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 347.716321] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 347.719528] Lustre: Skipped 1 previous similar message [ 347.775385] Lustre: lustre-MDT0001: new disk, initializing [ 347.851277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 347.881876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 347.898229] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 351.357500] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 357.720396] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 361.097397] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 366.490803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 366.716436] Lustre: lustre-OST0000: new disk, initializing [ 366.719971] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 366.728422] Lustre: 16692:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 366.802683] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 368.224833] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 368.235060] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 368.336869] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 371.914895] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 378.210515] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 382.003782] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 382.187769] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 385.923927] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 386.123355] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 389.099238] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 389.103929] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 389.117302] LustreError: Skipped 1 previous similar message [ 389.133752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 389.146172] Lustre: Skipped 1 previous similar message [ 393.932845] Lustre: server umount lustre-OST0000 complete [ 399.331306] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 399.335412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 399.340126] Lustre: Skipped 2 previous similar messages [ 399.359202] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 403.897459] Lustre: server umount lustre-MDT0000 complete [ 404.459958] LustreError: 14960:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 404.470066] LustreError: 14960:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 406.790770] LustreError: 16691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779870008 with bad export cookie 17663580181235705264 [ 406.802136] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 406.804636] LustreError: 16691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 407.020913] Lustre: server umount lustre-MDT0001 complete [ 411.651678] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 04:20:12 (1779870012) [ 426.753881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 427.090575] LustreError: 19419:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 427.167620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 430.701720] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 432.614649] LustreError: 19420:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 436.711236] LustreError: 19419:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 437.728755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 441.327191] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 445.489196] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 448.377658] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 450.013628] Lustre: Failing over lustre-MDT0000 [ 450.290061] Lustre: server umount lustre-MDT0000 complete [ 452.579505] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 452.587274] LustreError: 19414:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 452.593640] Lustre: Skipped 1 previous similar message [ 452.613385] LustreError: 19414:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 467.377830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 467.548602] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 470.905474] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 473.058689] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 473.065420] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 473.081972] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 477.222383] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 482.468458] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 482.478623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 482.481287] Lustre: Skipped 2 previous similar messages [ 486.614579] Lustre: server umount lustre-MDT0000 complete [ 487.905180] LustreError: 20487:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 487.926629] LustreError: 20487:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 489.593235] LustreError: 20760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779870091 with bad export cookie 17663580181235706678 [ 489.599597] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 489.604736] LustreError: 20760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 489.961940] Lustre: server umount lustre-MDT0001 complete [ 497.142469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 497.581820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 497.588163] Lustre: Skipped 2 previous similar messages [ 500.796304] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 507.754238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 510.768112] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 514.568562] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 516.642557] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 517.849057] Lustre: Failing over lustre-MDT0000 [ 517.969804] Lustre: server umount lustre-MDT0000 complete [ 518.113163] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 518.120094] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 518.130667] Lustre: Skipped 1 previous similar message [ 522.721751] LustreError: 22412:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 522.731523] LustreError: 22412:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 533.983914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 534.064046] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 536.834115] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 539.619182] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 539.621083] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 539.634812] Lustre: Skipped 1 previous similar message [ 539.643271] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 542.002754] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 548.839498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 548.854519] Lustre: Skipped 1 previous similar message [ 551.149855] Lustre: server umount lustre-MDT0000 complete [ 554.370972] LustreError: 22393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779870156 with bad export cookie 17663580181235707616 [ 554.376415] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 554.378262] LustreError: 22393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 554.613901] Lustre: server umount lustre-MDT0001 complete [ 560.745692] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 04:22:41 (1779870161) [ 573.100986] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [ 579.913963] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 605.509239] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 612.287970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 612.427311] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 612.449608] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 612.498919] Lustre: lustre-MDT0000: new disk, initializing [ 612.566271] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 614.960578] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 622.202433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 622.268899] Lustre: 28399:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 622.447553] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 624.529980] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 629.369375] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 631.710889] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 635.647118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 635.852493] Lustre: 30141:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 635.920907] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 635.924630] Lustre: Skipped 4 previous similar messages [ 637.222257] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 637.259331] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 637.267478] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 638.998858] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 643.391851] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 648.636270] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 663.012496] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 663.017892] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 663.022401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 663.023721] Lustre: Skipped 4 previous similar messages [ 663.028318] LustreError: Skipped 1 previous similar message [ 675.296223] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 675.410709] Lustre: server umount lustre-OST0000 complete [ 692.193692] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 692.319191] Lustre: server umount lustre-MDT0000 complete [ 693.729438] LustreError: 28412:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 693.738808] LustreError: 28412:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 694.861345] LustreError: 28393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779870296 with bad export cookie 17663580181235708015 [ 694.868427] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 694.875031] LustreError: 28393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 695.231724] Lustre: server umount lustre-MDT0001 complete [ 702.642348] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [ 704.749795] Key type lgssc unregistered [ 705.006835] LNet: 31986:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 705.011873] LNetError: 31986:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 705.029585] LNet: Removed LNI 192.168.204.156@tcp [ 705.778726] Key type .llcrypt unregistered [ 705.791336] Key type ._llcrypt unregistered [ 720.883598] Key type ._llcrypt registered [ 720.885900] Key type .llcrypt registered [ 720.991886] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 721.605675] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 721.651707] alg: No test for adler32 (adler32-zlib) [ 722.633119] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 722.786598] LNet: Added LNI 192.168.204.156@tcp [8/256/0/180] [ 724.424171] Key type lgssc registered [ 725.289662] Lustre: Echo OBD driver; http://www.lustre.org/ [ 731.053566] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 731.071801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 732.463465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 735.469816] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 741.046782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 741.274171] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 744.040471] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 747.691138] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 749.989206] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 754.099683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 754.291761] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 757.376883] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 760.491372] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 765.559481] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 773.983753] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 04:26:15 (1779870375) [ 775.323663] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 776.536605] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 780.679526] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 04:26:21 (1779870381) [ 786.069710] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 04:26:27 (1779870387) [ 786.714676] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 792.166244] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 792.170357] Lustre: Skipped 2 previous similar messages [ 793.359109] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 04:26:34 (1779870394) [ 798.663912] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 798.668172] Lustre: Skipped 4 previous similar messages [ 808.005638] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 808.011504] Lustre: Skipped 5 previous similar messages [ 809.995621] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 04:26:50 (1779870410) [ 811.936174] Lustre: *** cfs_fail_loc=131b, val=2*** [ 814.604904] Lustre: *** cfs_fail_loc=131b, val=2*** [ 820.558632] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 04:27:01 (1779870421) [ 824.070242] Lustre: Setting parameter general.jobid_name=TESTNAME1 in log params [ 824.075524] Lustre: Skipped 5 previous similar messages [ 831.200803] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 04:27:12 (1779870432) [ 840.674816] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 840.679063] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 840.686336] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 842.725145] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 842.725402] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 842.726610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 846.348424] Lustre: server umount lustre-OST0000 complete [ 852.963820] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 852.971911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 852.984356] Lustre: Skipped 1 previous similar message [ 852.986755] Lustre: Skipped 1 previous similar message [ 855.269824] Lustre: server umount lustre-MDT0000 complete [ 857.893629] LustreError: 33340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779870459 with bad export cookie 8099889570313897972 [ 857.898096] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 857.906840] LustreError: 33340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 858.137660] Lustre: server umount lustre-MDT0001 complete [ 864.403947] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [ 866.516347] Key type lgssc unregistered [ 866.793384] LNet: 40835:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 866.800371] LNetError: 40835:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 866.811573] LNet: Removed LNI 192.168.204.156@tcp [ 867.304548] Key type .llcrypt unregistered [ 867.307767] Key type ._llcrypt unregistered [ 878.331733] Key type ._llcrypt registered [ 878.333563] Key type .llcrypt registered [ 878.402231] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [ 887.636958] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 888.322057] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 888.344288] alg: No test for adler32 (adler32-zlib) [ 889.345843] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 889.507859] LNet: Added LNI 192.168.204.156@tcp [8/256/0/180] [ 891.144224] Key type lgssc registered [ 892.039598] Lustre: Echo OBD driver; http://www.lustre.org/ [ 914.148185] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 914.161347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 915.298591] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 915.319788] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 915.366916] Lustre: lustre-MDT0000: new disk, initializing [ 915.402472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 915.415740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 917.427490] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 924.310713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 924.384281] Lustre: 43935:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 924.429227] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 924.432105] Lustre: Skipped 1 previous similar message [ 924.475891] Lustre: lustre-MDT0001: new disk, initializing [ 924.524882] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 924.547731] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 924.553956] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 927.163898] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 932.018755] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 934.525919] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 938.837757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 939.038254] Lustre: lustre-OST0000: new disk, initializing [ 939.043726] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 939.053301] Lustre: 45677:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 939.137066] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 943.031476] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 945.689079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 945.696407] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 945.765230] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 947.714545] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 950.246661] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 950.359746] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 952.630225] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 952.749078] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 955.873671] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 955.878721] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 955.881059] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 955.891316] Lustre: Skipped 1 previous similar message [ 960.996309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 961.001159] Lustre: Skipped 2 previous similar messages [ 966.121096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 966.125762] Lustre: Skipped 1 previous similar message [ 968.160136] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 968.253368] Lustre: server umount lustre-OST0000 complete [ 970.722086] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 970.725917] LustreError: Skipped 1 previous similar message [ 970.727993] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 970.735592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 976.355236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 976.361140] Lustre: Skipped 3 previous similar messages [ 976.513587] Lustre: server umount lustre-MDT0000 complete [ 978.734227] LustreError: 43925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779870580 with bad export cookie 7680026921745844540 [ 978.737354] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 978.740271] LustreError: 43925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 978.966155] Lustre: server umount lustre-MDT0001 complete [ 984.613118] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 04:29:45 (1779870585) [ 988.278581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 988.489296] LustreError: 47988:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 988.500177] LustreError: 47988:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 988.535781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 990.798991] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 993.763386] LustreError: 47989:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 996.102435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 996.330001] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 998.623764] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1001.846515] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1004.095446] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1008.438978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1008.648572] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1011.937396] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1014.883805] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1021.848608] LustreError: 50175:0:(llog.c:444:llog_init_handle()) llog type is not specified! [ 1022.382240] Lustre: Modifying parameter general.timeout=1 in log params [ 1023.456445] Lustre: Modifying parameter general.timeout=3 in log params [ 1023.461900] Lustre: Skipped 1 previous similar message [ 1025.893467] Lustre: Modifying parameter general.timeout=8 in log params [ 1025.899815] Lustre: Skipped 4 previous similar messages [ 1030.355419] Lustre: Modifying parameter general.timeout=16 in log params [ 1030.359358] Lustre: Skipped 7 previous similar messages [ 1038.738045] Lustre: Modifying parameter general.timeout=30 in log params [ 1038.742146] Lustre: Skipped 13 previous similar messages [ 1054.748162] Lustre: Modifying parameter general.timeout=58 in log params [ 1054.751432] Lustre: Skipped 27 previous similar messages [ 1087.099489] Lustre: Modifying parameter general.timeout=93 in log params [ 1087.103777] Lustre: Skipped 34 previous similar messages [ 1115.856405] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 04:31:57 (1779870717) [ 1117.809632] Lustre: 57098:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1118.819494] Lustre: 57196:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1121.110706] Lustre: 57393:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1121.119866] Lustre: 57393:0:(mgs_llog.c:1437:mgs_modify_param()) Skipped 1 previous similar message [ 1138.775877] Lustre: 58963:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1138.782686] Lustre: 58963:0:(mgs_llog.c:1437:mgs_modify_param()) Skipped 2 previous similar messages [ 1138.792896] LustreError: 58963:0:(mgs_handler.c:1202:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1140.960440] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 04:32:22 (1779870742) [ 1148.901541] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1148.909068] Lustre: Skipped 2 previous similar messages [ 1148.912088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1148.918603] Lustre: Skipped 2 previous similar messages [ 1154.455937] Lustre: server umount lustre-MDT0000 complete [ 1156.688155] LustreError: 50225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779870758 with bad export cookie 7680026921745845422 [ 1156.694791] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1156.699031] LustreError: 50225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1156.858419] Lustre: server umount lustre-MDT0001 complete [ 1168.538871] Lustre: server umount lustre-OST0000 complete [ 1183.349615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1183.520772] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1183.539856] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1183.545185] Lustre: Skipped 59 previous similar messages [ 1183.677737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1185.696229] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1190.668525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1190.763073] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1190.781626] Lustre: 61490:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1193.515530] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1197.389120] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1199.997784] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1204.701927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1204.859834] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1208.971096] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1212.553719] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1219.182278] Lustre: 63715:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1228.257520] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1228.260470] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1228.263727] LustreError: Skipped 1 previous similar message [ 1228.268926] Lustre: Skipped 1 previous similar message [ 1228.271411] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1228.282958] Lustre: Skipped 4 previous similar messages [ 1229.426844] Lustre: server umount lustre-OST0000 complete [ 1231.841447] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1233.380901] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1233.389513] Lustre: Skipped 2 previous similar messages [ 1237.702079] Lustre: server umount lustre-MDT0000 complete [ 1238.501791] LustreError: 62669:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1238.515631] LustreError: 62669:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1240.025329] LustreError: 61480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779870841 with bad export cookie 7680026921745848096 [ 1240.033256] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1240.192952] Lustre: server umount lustre-MDT0001 complete [ 1246.577859] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [ 1248.248945] Key type lgssc unregistered [ 1248.419818] LNet: 64903:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1248.423353] LNetError: 64903:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1248.432901] LNet: Removed LNI 192.168.204.156@tcp [ 1248.863120] Key type .llcrypt unregistered [ 1248.864593] Key type ._llcrypt unregistered [ 1262.989616] Key type ._llcrypt registered [ 1262.991576] Key type .llcrypt registered [ 1263.041812] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 1263.509885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1263.564949] alg: No test for adler32 (adler32-zlib) [ 1264.528688] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 1264.690935] LNet: Added LNI 192.168.204.156@tcp [8/256/0/180] [ 1266.304256] Key type lgssc registered [ 1266.927403] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1271.113790] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1271.125636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1272.411249] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1274.576626] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1279.389415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1279.747863] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1281.788869] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1285.348667] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1287.598065] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1291.333204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1291.731850] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1294.962343] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1297.955536] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1311.849705] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1318.455895] Lustre: server umount lustre-MDT0000 complete [ 1320.296669] LustreError: 66256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779870921 with bad export cookie 13092793098623394425 [ 1320.301157] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1320.305895] LustreError: 66256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1320.418783] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1320.419718] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1320.423965] Lustre: Skipped 1 previous similar message [ 1320.424839] LustreError: 66272:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1320.438976] LustreError: 66272:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1322.527476] Lustre: server umount lustre-MDT0001 complete [ 1324.577585] Lustre: server umount lustre-OST0000 complete [ 1338.789422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1338.943238] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1338.964088] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1339.133690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1341.193924] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1345.835159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1345.913904] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1345.936542] Lustre: 70823:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1346.194682] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1348.041820] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1350.869297] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1352.850160] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1356.473365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1356.595735] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1356.823686] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1359.676511] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1362.226111] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1362.968498] Lustre: 73056:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1363.872203] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1367.522366] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1367.525920] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1367.532590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1367.535927] Lustre: Skipped 1 previous similar message [ 1372.031311] Lustre: server umount lustre-OST0000 complete [ 1376.737891] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1376.742351] LustreError: Skipped 1 previous similar message [ 1376.746169] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1376.755565] Lustre: Skipped 1 previous similar message [ 1376.760452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1380.425436] Lustre: server umount lustre-MDT0000 complete [ 1382.320203] LustreError: 71549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779870983 with bad export cookie 13092793098623395244 [ 1382.322325] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1382.326472] LustreError: 71549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1382.474419] Lustre: server umount lustre-MDT0001 complete [ 1387.485512] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [ 1388.970705] Key type lgssc unregistered [ 1389.146942] LNet: 74245:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1389.152807] LNetError: 74245:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1390.182826] LNet: Removed LNI 192.168.204.156@tcp [ 1390.579675] Key type .llcrypt unregistered [ 1390.581162] Key type ._llcrypt unregistered [ 1402.978455] Key type ._llcrypt registered [ 1402.980183] Key type .llcrypt registered [ 1403.030248] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 1403.438201] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1403.489267] alg: No test for adler32 (adler32-zlib) [ 1404.378599] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 1404.478392] LNet: Added LNI 192.168.204.156@tcp [8/256/0/180] [ 1406.074197] Key type lgssc registered [ 1406.587388] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1410.140070] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1410.153567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1411.395373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1413.031244] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1417.224271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1417.514034] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1419.295313] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1421.696791] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1423.196872] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1426.590186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1426.902140] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1429.961521] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1432.743439] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1440.428298] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1441.297932] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1441.301971] Lustre: Skipped 1 previous similar message [ 1442.731942] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1442.735128] Lustre: Skipped 2 previous similar messages [ 1444.882551] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1444.884944] Lustre: Skipped 4 previous similar messages [ 1449.008807] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1449.011181] Lustre: Skipped 8 previous similar messages [ 1457.412491] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1457.419738] Lustre: Skipped 16 previous similar messages [ 1473.485048] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1473.487589] Lustre: Skipped 39 previous similar messages [ 1486.848144] Lustre: 83355:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1488.223559] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 04:38:09 (1779871089) [ 1489.080480] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1489.896701] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 04:38:11 (1779871091) [ 1492.451471] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1492.451638] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1492.461603] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1492.471256] Lustre: Skipped 1 previous similar message [ 1497.570408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1497.575164] Lustre: Skipped 1 previous similar message [ 1497.638405] Lustre: server umount lustre-OST0000 complete [ 1499.617034] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1499.620897] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1499.626207] Lustre: Skipped 1 previous similar message [ 1499.628588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1502.689863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1502.690621] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1505.583443] Lustre: server umount lustre-MDT0000 complete [ 1507.463060] LustreError: 77867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779871109 with bad export cookie 12998170600554922009 [ 1507.464848] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1507.470340] LustreError: 77867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1507.625665] Lustre: server umount lustre-MDT0001 complete [ 1512.052811] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [ 1513.382703] Key type lgssc unregistered [ 1513.532418] LNet: 84583:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1513.536235] LNetError: 84583:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1513.545486] LNet: Removed LNI 192.168.204.156@tcp [ 1513.932232] Key type .llcrypt unregistered [ 1513.933748] Key type ._llcrypt unregistered [ 1515.724732] Key type ._llcrypt registered [ 1515.726341] Key type .llcrypt registered [ 1515.780336] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1517.663950] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules [ 1518.211754] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1518.224842] alg: No test for adler32 (adler32-zlib) [ 1519.117460] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 1519.222731] LNet: Added LNI 192.168.204.156@tcp [8/256/0/180] [ 1520.816225] Key type lgssc registered [ 1521.363715] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1521.857732] LustreError: 85704:0:(lu_object.c:1479:lu_context_key_register()) cfs_race id 60d sleeping [ 1531.950729] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 1536.341720] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1543.033562] LustreError: 85704:0:(lu_object.c:1479:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1543.047101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1544.328375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1545.949722] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1549.915088] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 04:39:11 (1779871151) [ 1552.029913] LustreError: 86743:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1552.035362] LustreError: 86743:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1552.039529] LustreError: 86743:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 0, failed: rc = -5 [ 1552.173583] Lustre: server umount lustre-MDT0000 complete [ 1558.039744] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [ 1559.315214] Key type lgssc unregistered [ 1559.474418] LNet: 88199:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1559.478199] LNetError: 88199:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1559.488564] LNet: Removed LNI 192.168.204.156@tcp [ 1559.869384] Key type .llcrypt unregistered [ 1559.871280] Key type ._llcrypt unregistered [ 1568.654377] Key type ._llcrypt registered [ 1568.655870] Key type .llcrypt registered [ 1568.706291] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 1569.098383] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1569.161106] alg: No test for adler32 (adler32-zlib) [ 1570.080906] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 1570.215341] LNet: Added LNI 192.168.204.156@tcp [8/256/0/180] [ 1571.816212] Key type lgssc registered [ 1572.417407] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1576.622226] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1576.644732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1578.041438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1580.279893] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1585.107969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1585.446649] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1587.448283] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1590.525863] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1592.476684] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1595.897282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1596.201885] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1598.981540] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1601.534704] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1636.801851] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 04:40:38 (1779871238) [ 1637.498865] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1638.247256] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 04:40:39 (1779871239) [ 1640.420588] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1640.421489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1640.426073] Lustre: Skipped 1 previous similar message [ 1645.528048] Lustre: server umount lustre-MDT0000 complete [ 1645.538426] LustreError: 90093:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1645.543937] LustreError: 90093:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1646.975355] LustreError: 89360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779871248 with bad export cookie 12385567898690462454 [ 1646.978501] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1646.981735] LustreError: 89360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1647.121117] Lustre: server umount lustre-MDT0001 complete [ 1658.962405] Lustre: server umount lustre-OST0000 complete [ 1663.930363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1664.158117] LustreError: 93330:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1664.244800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1666.072161] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1669.602837] LustreError: 93329:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1669.994492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1670.331075] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1672.137092] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1674.963527] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1676.729263] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1682.723270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1682.828780] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1682.833438] LustreError: 93318:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1682.847682] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1682.852215] LustreError: 95185:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -98 [ 1682.855920] LustreError: 95185:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 1682.858941] LustreError: 95185:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1682.895371] Lustre: server umount lustre-OST0000 complete [ 1682.897430] LustreError: 95185:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1686.719499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1686.816815] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1686.820313] LustreError: 93318:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1686.824303] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1686.827625] LustreError: 95650:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -98 [ 1686.830395] LustreError: 95650:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 1686.832134] LustreError: 95650:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1686.859264] Lustre: server umount lustre-OST0000 complete [ 1686.860790] LustreError: 95650:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1691.263129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1691.352518] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1691.355113] Lustre: Found index 0 for lustre-OST0000, updating log [ 1691.357794] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1691.391122] Lustre: lustre-OST0000: new disk, initializing [ 1691.393697] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1691.397802] Lustre: 96183:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1691.532099] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1694.291997] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1695.947967] Lustre: Failing over lustre-OST0000 [ 1696.023768] Lustre: server umount lustre-OST0000 complete [ 1699.810060] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1699.810799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1699.817737] Lustre: Skipped 2 previous similar messages [ 1699.820691] Lustre: Skipped 2 previous similar messages [ 1703.836469] Lustre: server umount lustre-MDT0000 complete [ 1704.929508] LustreError: 93329:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1704.938524] LustreError: 93329:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1705.423215] LustreError: 93310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779871307 with bad export cookie 12385567898690465184 [ 1705.427103] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1705.428777] LustreError: 93310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1715.751321] Lustre: server umount lustre-MDT0001 complete [ 1719.193704] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 04:42:00 (1779871320) [ 1725.857851] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 1729.960322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1730.134028] LustreError: 98492:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1730.170253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1731.866147] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1735.163856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1736.786365] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1737.865571] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 1740.190463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1740.444437] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 1742.489164] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1744.548122] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1744.559978] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 1744.560156] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1745.690469] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1745.720823] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 1745.756367] Lustre: lustre-OST0001: new disk, initializing [ 1745.758734] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1745.761980] Lustre: 100756:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1748.278354] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1748.564053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1748.570854] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1748.613397] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1752.009147] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1757.097873] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1759.612490] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.56@tcp (stopping) [ 1760.225663] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 1760.230203] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1760.242537] Lustre: Skipped 2 previous similar messages [ 1764.715459] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.56@tcp (stopping) [ 1764.718409] Lustre: Skipped 3 previous similar messages [ 1769.837146] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.56@tcp (stopping) [ 1769.840384] Lustre: Skipped 3 previous similar messages [ 1773.024111] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1773.084401] Lustre: server umount lustre-MDT0001 complete [ 1774.955235] LustreError: 99825:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1776.442374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1776.478714] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1776.480814] Lustre: Found index 1 for lustre-MDT0001, updating log [ 1776.496232] Lustre: 98481:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1776.723120] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1776.726657] Lustre: Skipped 3 previous similar messages [ 1777.988796] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 1778.469926] LustreError: 102568:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 1778.472632] LustreError: 102568:0:(tgt_mount.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 1778.475487] LustreError: 98484:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 1778.501269] LustreError: 102575:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.156@tcp: cfg command failed: rc = -17 [ 1778.504592] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1778.504592] [ 1780.976972] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 04:43:02 (1779871382) [ 1781.734602] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1786.634934] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 1792.945920] Lustre: 104318:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1802.983265] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1806.101829] Lustre: 105451:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1945.573907] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1945.577942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1945.581982] Lustre: Skipped 1 previous similar message [ 1945.587448] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1945.588839] Lustre: Skipped 5 previous similar messages [ 1947.204567] Lustre: server umount lustre-MDT0000 complete [ 1950.711531] LustreError: 98488:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1950.725922] LustreError: 98488:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 1951.147578] LustreError: 99215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779871552 with bad export cookie 12385567898690466101 [ 1951.155368] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1951.164875] LustreError: 99215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1951.777222] Lustre: server umount lustre-MDT0001 complete [ 1966.994855] Lustre: server umount lustre-OST0000 complete [ 1981.425505] Lustre: server umount lustre-OST0001 complete [ 1987.646517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1997.217221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2006.505936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2016.335788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2035.327115] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 2045.564126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2045.588747] Lustre: lustre-MDT0000: reset Object Index mappings [ 2045.968262] LustreError: 109409:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2046.102390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2050.344634] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 2059.179122] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2059.196954] Lustre: lustre-MDT0001: reset Object Index mappings [ 2064.092153] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 2067.009844] Lustre: 110523:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2073.601793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2080.463965] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 2084.667199] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:227 to 0x280000400:257) [ 2090.158887] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2091.962850] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 2091.982892] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 2092.076684] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:354 to 0x280000401:386) [ 2097.951080] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 2106.362246] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2110.495629] Lustre: 112364:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2169.733549] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 04:49:30 (1779871770) [ 2197.986052] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2198.002152] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2198.016245] Lustre: Skipped 2 previous similar messages [ 2198.026655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2198.029330] Lustre: Skipped 1 previous similar message [ 2201.988233] Lustre: server umount lustre-MDT0000 complete [ 2204.651980] LustreError: 110129:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2204.667944] LustreError: 110129:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 2205.677836] LustreError: 109390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779871807 with bad export cookie 12385567898690512938 [ 2205.688743] LustreError: 109390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2205.693873] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2209.762112] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2209.780321] Lustre: Skipped 4 previous similar messages [ 2212.226367] Lustre: server umount lustre-MDT0001 complete [ 2222.299563] Lustre: server umount lustre-OST0000 complete [ 2232.647420] Lustre: server umount lustre-OST0001 complete [ 2239.892699] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [ 2252.132784] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 2290.573574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2290.799216] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2290.829169] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2290.896425] Lustre: lustre-MDT0000: new disk, initializing [ 2290.998200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2291.012528] Lustre: Skipped 3 previous similar messages [ 2291.035644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2295.905902] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 2303.437853] Lustre: server umount lustre-MDT0000 complete [ 2313.939354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2314.336663] Lustre: Found index 0 for lustre-MDT0000, updating log [ 2314.350239] Lustre: 116594:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2314.364316] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 2320.069755] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 2325.395783] Lustre: server umount lustre-MDT0000 complete [ 2334.213877] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 04:52:15 (1779871935) [ 2336.057812] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 2338.000758] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 04:52:18 (1779871938) [ 2355.907711] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 2365.488051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2370.159223] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 2379.269767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2379.455277] Lustre: 118609:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2379.522216] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2379.525488] Lustre: Skipped 1 previous similar message [ 2379.613181] Lustre: lustre-MDT0001: new disk, initializing [ 2379.763068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2379.781415] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2383.932601] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 2388.940464] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2396.008309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2396.259685] Lustre: lustre-OST0000: new disk, initializing [ 2396.263868] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2396.270411] Lustre: 120153:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2397.606818] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2397.617342] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2397.737715] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2402.726975] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 2413.856596] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2414.051703] Lustre: lustre-OST0001: new disk, initializing [ 2414.057704] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2414.065663] Lustre: 121009:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2415.934525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2415.948467] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2416.040931] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2420.702441] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 2429.788850] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2432.861245] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2841.158242] Lustre: 127408:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 2841.169235] Lustre: 127408:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 2841.181227] Lustre: 127408:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 2841.191392] Lustre: 127408:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 2841.199392] Lustre: 127408:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 2841.212269] Lustre: 127408:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3187.068948] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 05:06:26 (1779872786) [ 3204.579792] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3204.588113] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3204.605632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3204.624889] Lustre: Skipped 3 previous similar messages [ 3204.652156] Lustre: Skipped 11 previous similar messages [ 3208.511931] Lustre: server umount lustre-MDT0000 complete [ 3209.699239] LustreError: 121808:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3209.725784] LustreError: 121808:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 3212.208747] LustreError: 127446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779872813 with bad export cookie 12385567898690583036 [ 3212.219876] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3212.228303] LustreError: 127446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3214.822742] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3214.837038] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3214.850438] Lustre: Skipped 1 previous similar message [ 3214.871592] Lustre: Skipped 1 previous similar message [ 3218.743337] Lustre: server umount lustre-MDT0001 complete [ 3237.346329] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3237.485717] Lustre: server umount lustre-OST0000 complete [ 3256.288223] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3256.651499] Lustre: server umount lustre-OST0001 complete [ 3264.543157] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [ 3276.292353] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 3319.516875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3319.811807] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3319.850737] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3319.938792] Lustre: lustre-MDT0000: new disk, initializing [ 3320.076125] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3320.081556] Lustre: Skipped 5 previous similar messages [ 3320.109572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3325.356494] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 3337.246398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3337.387823] Lustre: 130462:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3337.461532] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3337.465589] Lustre: Skipped 1 previous similar message [ 3337.521555] Lustre: lustre-MDT0001: new disk, initializing [ 3337.691008] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3337.695727] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3342.610968] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 3351.981963] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3356.456098] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3364.951029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3365.324834] Lustre: lustre-OST0000: new disk, initializing [ 3365.337305] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3365.348351] Lustre: 132202:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3365.466562] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3365.481137] Lustre: Skipped 1 previous similar message [ 3367.062537] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3367.081471] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3367.317785] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3372.778881] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 3381.724714] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3392.358408] Lustre: lustre-MDD0000: changelog on [ 3395.075933] Lustre: lustre-MDD0001: changelog on [ 5613.125899] Lustre: lustre-MDD0001: changelog off [ 5614.708442] Lustre: 130469:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5615.514807] Lustre: lustre-MDD0000: changelog off [ 5625.315234] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5625.319422] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5625.332742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5625.339585] Lustre: Skipped 1 previous similar message [ 5625.356614] Lustre: Skipped 3 previous similar messages [ 5626.937985] Lustre: server umount lustre-OST0000 complete [ 5631.458940] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5631.470749] LustreError: Skipped 1 previous similar message [ 5631.476758] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5631.495673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5635.557103] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5635.566097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5635.591308] Lustre: Skipped 1 previous similar message [ 5637.012192] Lustre: server umount lustre-MDT0000 complete [ 5640.664171] LustreError: 131238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779875242 with bad export cookie 12385567898691403534 [ 5640.673080] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5640.681247] LustreError: 131238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5641.253191] Lustre: server umount lustre-MDT0001 complete [ 5651.624686] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [ 5654.410703] Key type lgssc unregistered [ 5654.691959] LNet: 141266:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5654.699494] LNetError: 141266:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5654.725175] LNet: Removed LNI 192.168.204.156@tcp [ 5655.630318] Key type .llcrypt unregistered [ 5655.634627] Key type ._llcrypt unregistered [ 5675.381341] Key type ._llcrypt registered [ 5675.383464] Key type .llcrypt registered [ 5675.468790] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [ 5690.851508] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 5692.162843] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5692.225192] alg: No test for adler32 (adler32-zlib) [ 5693.297878] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 5693.561260] LNet: Added LNI 192.168.204.156@tcp [8/256/0/180] [ 5695.240524] Key type lgssc registered [ 5696.563915] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5730.121571] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5730.142511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5731.429457] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5731.507780] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5731.623226] Lustre: lustre-MDT0000: new disk, initializing [ 5731.784734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5731.826691] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5735.962829] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 5745.270226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5745.359719] Lustre: 144428:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5745.395624] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5745.398664] Lustre: Skipped 1 previous similar message [ 5745.441851] Lustre: lustre-MDT0001: new disk, initializing [ 5745.522790] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5745.554714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5745.566733] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5749.972598] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 5757.466072] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5761.263677] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5768.164431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5768.491181] Lustre: lustre-OST0000: new disk, initializing [ 5768.494630] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5768.498674] Lustre: 146175:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5768.577186] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5775.466441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5775.481837] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5775.622662] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5775.802942] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 5784.262691] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5790.492089] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_module obdecho/obdecho [ 5793.169078] LustreError: 147049:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 5793.186918] LustreError: 147049:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 5795.549636] LustreError: 147145:0:(echo_client.c:1883:echo_md_handler()) Can not resolve the path : rc = -2 [ 5811.690041] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5811.699908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5811.702746] Lustre: Skipped 2 previous similar messages [ 5811.726677] Lustre: Skipped 2 previous similar messages [ 5815.767821] Lustre: server umount lustre-MDT0000 complete [ 5819.647625] LustreError: 144422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779875421 with bad export cookie 10090135620323985746 [ 5819.657295] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5819.666637] LustreError: 144422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5819.998090] Lustre: server umount lustre-MDT0001 complete [ 5834.304186] Lustre: server umount lustre-OST0000 complete [ 5857.092908] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [ 5867.210913] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 5906.606244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5906.733634] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5906.776959] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5906.875964] Lustre: lustre-MDT0000: new disk, initializing [ 5906.975397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5906.992782] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5910.274821] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 5920.482263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5920.622930] Lustre: 150828:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5920.717951] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5920.721771] Lustre: Skipped 1 previous similar message [ 5920.799917] Lustre: lustre-MDT0001: new disk, initializing [ 5920.902541] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5920.967984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5920.977191] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5924.810593] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 5932.642332] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5936.708255] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5944.245576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5944.538515] Lustre: lustre-OST0000: new disk, initializing [ 5944.542263] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5944.552390] Lustre: 152567:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5944.639749] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5950.932962] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 5953.106600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5953.115766] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5953.223119] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5958.160965] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5962.269910] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5962.462547] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5966.219076] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5966.411906] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5968.866175] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5968.874799] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5968.891866] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5968.900849] Lustre: Skipped 1 previous similar message [ 5973.989225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5973.993677] Lustre: Skipped 1 previous similar message [ 5974.187631] Lustre: server umount lustre-OST0000 complete [ 5979.105700] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5979.106593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5979.117449] Lustre: Skipped 2 previous similar messages [ 5979.126271] Lustre: Skipped 1 previous similar message [ 5983.846440] Lustre: server umount lustre-MDT0000 complete [ 5984.229160] LustreError: 150836:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5984.247729] LustreError: 150836:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5987.076893] LustreError: 150819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779875588 with bad export cookie 10090135620323986705 [ 5987.088203] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5987.090630] LustreError: 150819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5987.343088] Lustre: server umount lustre-MDT0001 complete [ 5990.540769] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 05:53:11 (1779875591) [ 5991.853538] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 5993.347354] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 05:53:14 (1779875594) [ 5999.910459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6000.520401] LustreError: 154834:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6000.624335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6004.830946] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 6005.735471] LustreError: 154835:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6010.859267] LustreError: 154834:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6014.692784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6020.175648] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 6025.771677] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6028.767106] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6034.281852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6034.475176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6034.480020] Lustre: Skipped 1 previous similar message [ 6039.558583] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 6044.472779] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6055.910650] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6055.920509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6055.928498] Lustre: Skipped 2 previous similar messages [ 6055.948229] Lustre: Skipped 2 previous similar messages [ 6061.346974] Lustre: server umount lustre-MDT0000 complete [ 6065.426085] LustreError: 154816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779875667 with bad export cookie 10090135620323987580 [ 6065.443614] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6065.521690] LustreError: 154830:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6065.550320] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.56@tcp (stopping) [ 6065.562200] Lustre: Skipped 4 previous similar messages [ 6066.145703] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6071.818452] Lustre: server umount lustre-MDT0001 complete [ 6077.393877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6090.464942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6090.673056] LustreError: 158336:0:(llog.c:1646:llog_backup()) MGC192.168.204.156@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6090.684990] Lustre: 158336:0:(mgc_request_server.c:768:mgc_llog_local_copy()) MGC192.168.204.156@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6094.759732] LustreError: 158341:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6094.781849] LustreError: 158341:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6111.603725] LustreError: 158358:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6111.623469] LustreError: 158358:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 16 previous similar messages [ 6146.420202] LustreError: 158358:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6146.439610] LustreError: 158358:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 24 previous similar messages [ 6152.160394] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6152.167788] Lustre: 158336:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.156@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6193.146658] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6197.447965] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 6198.250798] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6206.761925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6211.555616] LustreError: 142654:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f995739e x1866334517454976/t0(0) o250->MGC192.168.204.156@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6211.584053] LustreError: 158341:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6211.614160] LustreError: 158341:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 46 previous similar messages [ 6216.590627] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 6217.203847] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6222.319934] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6223.763402] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6227.429539] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6228.976287] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6247.905988] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6247.907885] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6247.914155] LustreError: Skipped 1 previous similar message [ 6247.917787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6247.929261] Lustre: Skipped 2 previous similar messages [ 6247.931628] Lustre: Skipped 4 previous similar messages [ 6253.576718] Lustre: server umount lustre-MDT0000 complete [ 6257.735743] LustreError: 154814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779875859 with bad export cookie 10090135620323994041 [ 6257.750538] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6257.752034] LustreError: 154814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 6258.096939] Lustre: server umount lustre-MDT0001 complete [ 6272.254540] Lustre: server umount lustre-OST0000 complete [ 6293.945488] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [ 6305.329140] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 6342.031377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6342.202707] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6342.232644] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6342.305529] Lustre: lustre-MDT0000: new disk, initializing [ 6342.369640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6342.372644] Lustre: Skipped 1 previous similar message [ 6342.385379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6345.365784] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 6355.728443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6355.883502] Lustre: 163179:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6356.200639] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6360.564872] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 6369.140533] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6373.866936] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6381.737175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6381.953498] Lustre: lustre-OST0000: new disk, initializing [ 6381.956730] Lustre: Skipped 1 previous similar message [ 6381.959547] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6381.967954] Lustre: Skipped 2 previous similar messages [ 6381.975658] Lustre: 164918:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6387.789966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6387.811673] Lustre: Skipped 1 previous similar message [ 6387.822883] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6387.974788] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6388.669867] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 6396.595857] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6401.138438] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6401.421052] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6405.337199] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6405.579878] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6408.675888] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6408.677035] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6408.685285] LustreError: Skipped 1 previous similar message [ 6408.686595] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6408.701736] Lustre: Skipped 1 previous similar message [ 6408.722237] Lustre: Skipped 5 previous similar messages [ 6413.445405] Lustre: server umount lustre-OST0000 complete [ 6417.889162] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6431.200595] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6431.487718] Lustre: server umount lustre-MDT0000 complete [ 6434.273935] LustreError: 163190:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6434.286294] LustreError: 163190:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6435.328829] LustreError: 163173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779876036 with bad export cookie 10090135620324010869 [ 6435.340641] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6435.345239] LustreError: 163173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6435.721195] Lustre: server umount lustre-MDT0001 complete [ 6447.028131] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 06:00:46 (1779876046) [ 6455.638894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6461.881027] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 6472.874260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6478.124646] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 6484.722663] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6489.669817] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6496.950807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6504.061487] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 6510.415321] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6520.940831] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 06:02:01 (1779876121) [ 6527.972642] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6527.976491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6527.997817] Lustre: Skipped 3 previous similar messages [ 6528.003291] Lustre: Skipped 7 previous similar messages [ 6531.515104] Lustre: server umount lustre-OST0000 complete [ 6542.133871] Lustre: server umount lustre-MDT0000 complete [ 6545.790458] LustreError: 167166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779876147 with bad export cookie 10090135620324011751 [ 6545.795247] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6545.825504] LustreError: 167166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6546.280247] Lustre: server umount lustre-MDT0001 complete [ 6559.830809] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [ 6563.671599] Key type lgssc unregistered [ 6564.057069] LNet: 170569:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6564.082439] LNetError: 170569:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6564.106607] LNet: Removed LNI 192.168.204.156@tcp [ 6564.911247] Key type .llcrypt unregistered [ 6564.913610] Key type ._llcrypt unregistered [ 6591.933385] Key type ._llcrypt registered [ 6591.937236] Key type .llcrypt registered [ 6592.086966] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 6593.540058] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6593.579082] alg: No test for adler32 (adler32-zlib) [ 6594.620618] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 6594.954354] LNet: Added LNI 192.168.204.156@tcp [8/256/0/180] [ 6596.720189] Key type lgssc registered [ 6598.003417] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6607.839811] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6607.882824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6623.521305] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6628.640586] LustreError: 171797:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.156@tcp: failed processing log, type 4: rc = -110 [ 6633.764720] LustreError: 171797:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6633.784269] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 6633.790589] LustreError: MGC192.168.204.156@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 6633.802384] LustreError: 171797:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-OST0000: -22 [ 6633.814285] LustreError: 171797:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -22 [ 6633.819445] LustreError: 171797:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 6633.906466] Lustre: server umount lustre-OST0000 complete [ 6633.913126] LustreError: 171797:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6644.120690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6645.781308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6651.188501] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 6655.973202] LustreError: 172298:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6655.987782] LustreError: 172298:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6660.745884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6661.217152] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6666.925726] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 6673.671956] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6677.652934] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6685.479237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6685.680376] LustreError: 173893:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6685.817404] LustreError: 173888:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6685.834365] Lustre: 173888:0:(mgc_request_server.c:748:mgc_llog_local_copy()) MGC192.168.204.156@tcp: invalid local config log lustre-OST0000: rc = -22 [ 6686.058592] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6693.460703] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 6701.194072] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6706.662380] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6706.669477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6706.678100] Lustre: Skipped 2 previous similar messages [ 6706.687328] Lustre: Skipped 2 previous similar messages [ 6710.389805] Lustre: server umount lustre-MDT0000 complete [ 6711.779503] LustreError: 172297:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6711.805508] LustreError: 172297:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6714.252545] LustreError: 172277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779876315 with bad export cookie 12325215867544162151 [ 6714.259364] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6714.273981] LustreError: 172277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6714.802147] Lustre: server umount lustre-MDT0001 complete [ 6730.482968] Lustre: server umount lustre-OST0000 complete [ 6743.523954] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [ 6758.354736] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 6805.413126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6805.709830] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6805.779399] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6805.933688] Lustre: lustre-MDT0000: new disk, initializing [ 6806.034187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6806.055434] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6810.510927] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 6821.871743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6821.970872] Lustre: 177289:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6822.031150] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6822.037637] Lustre: Skipped 1 previous similar message [ 6822.099325] Lustre: lustre-MDT0001: new disk, initializing [ 6822.197195] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6822.240279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6822.258658] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6827.519380] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 6837.359485] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6842.197901] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6849.517081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6849.890959] Lustre: lustre-OST0000: new disk, initializing [ 6849.896281] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6849.902241] Lustre: 179032:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6850.065361] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6857.798711] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 6860.346175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6860.375425] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6860.495782] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6866.477417] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6873.491076] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6873.974438] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6879.661988] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6879.976334] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6886.378200] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6886.386072] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6886.407618] LustreError: Skipped 1 previous similar message [ 6886.419328] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6886.431339] Lustre: Skipped 1 previous similar message [ 6886.458304] Lustre: Skipped 1 previous similar message [ 6889.786807] Lustre: server umount lustre-OST0000 complete [ 6896.611512] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6896.616613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6896.633182] Lustre: Skipped 1 previous similar message [ 6896.649445] Lustre: Skipped 1 previous similar message [ 6901.166369] Lustre: server umount lustre-MDT0000 complete [ 6901.747494] LustreError: 177300:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6901.768723] LustreError: 177300:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6906.284410] LustreError: 177280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779876507 with bad export cookie 12325215867544162872 [ 6906.296783] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6906.309829] LustreError: 177280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6906.967747] Lustre: server umount lustre-MDT0001 complete [ 6919.887536] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 06:08:39 (1779876519) [ 6943.946718] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [ 6946.979931] Key type lgssc unregistered [ 6947.361152] LNet: 181814:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6947.373748] LNetError: 181814:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6947.400358] LNet: Removed LNI 192.168.204.156@tcp [ 6948.297556] Key type .llcrypt unregistered [ 6948.299607] Key type ._llcrypt unregistered [ 6967.875669] Key type ._llcrypt registered [ 6967.877398] Key type .llcrypt registered [ 6968.014741] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 6969.143205] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6969.227765] alg: No test for adler32 (adler32-zlib) [ 6970.347419] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 6970.675535] LNet: Added LNI 192.168.204.156@tcp [8/256/0/180] [ 6972.360316] Key type lgssc registered [ 6973.827785] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6982.678382] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6982.696202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6984.270304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6989.332772] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 6993.385088] LustreError: 183018:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6993.393706] LustreError: 183018:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6998.500696] LustreError: 183019:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6999.168648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6999.587834] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7004.114852] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 7010.552129] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7014.689472] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7023.159745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7023.472215] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7023.478484] LustreError: 184660:0:(llog.c:1687:llog_backup()) MGC192.168.204.156@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7023.496615] Lustre: 184660:0:(mgc_request_server.c:745:mgc_llog_local_copy()) MGC192.168.204.156@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7023.663252] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7030.609736] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 7037.064697] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7040.481876] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7040.504425] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7040.522336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7044.066349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7044.074970] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7044.111491] Lustre: Skipped 1 previous similar message [ 7046.122945] Lustre: server umount lustre-MDT0000 complete [ 7049.188801] LustreError: 183018:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7050.006684] LustreError: 182998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779876651 with bad export cookie 7933636864712547661 [ 7050.007916] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7050.026075] LustreError: 182998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7050.466203] Lustre: server umount lustre-MDT0001 complete [ 7066.535247] Lustre: server umount lustre-OST0000 complete [ 7077.948631] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [ 7089.900773] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 7133.858447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7134.226633] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7134.259132] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7134.317590] Lustre: lustre-MDT0000: new disk, initializing [ 7134.405431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7134.427367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7139.260437] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 7151.529625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7151.686707] Lustre: 188037:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7151.778857] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7151.782744] Lustre: Skipped 1 previous similar message [ 7151.792930] LustreError: 188802:0:(llog.c:1687:llog_backup()) MGC192.168.204.156@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7151.802869] LustreError: 188802:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 7151.808753] Lustre: 188802:0:(mgc_request_server.c:768:mgc_llog_local_copy()) MGC192.168.204.156@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7151.834710] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7151.848805] Lustre: Skipped 1 previous similar message [ 7151.917971] Lustre: lustre-MDT0001: new disk, initializing [ 7152.020829] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7152.062730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7152.075766] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7156.924753] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 7167.141790] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7172.376359] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7180.639982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7180.972901] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7180.977320] Lustre: Skipped 1 previous similar message [ 7180.986635] LustreError: 189757:0:(llog.c:1687:llog_backup()) MGC192.168.204.156@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7181.006652] LustreError: 189757:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 7181.029143] Lustre: 189757:0:(mgc_request_server.c:768:mgc_llog_local_copy()) MGC192.168.204.156@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7181.057733] Lustre: 189757:0:(mgc_request_server.c:768:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7181.114367] Lustre: lustre-OST0000: new disk, initializing [ 7181.124335] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7181.128732] Lustre: 189782:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7181.257650] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7187.706367] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 7190.589964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7190.610877] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7190.795392] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7196.453436] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7201.420839] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7201.599989] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7206.805882] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7207.077997] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7211.499521] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7211.509267] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7211.518216] LustreError: Skipped 1 previous similar message [ 7211.527673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7211.540114] Lustre: Skipped 1 previous similar message [ 7211.567087] Lustre: Skipped 3 previous similar messages [ 7215.802802] Lustre: server umount lustre-OST0000 complete [ 7221.730512] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7221.745027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7221.752105] Lustre: Skipped 1 previous similar message [ 7221.780027] Lustre: Skipped 1 previous similar message [ 7226.856098] LustreError: 188043:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7226.871340] LustreError: 188043:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7226.928657] Lustre: server umount lustre-MDT0000 complete [ 7231.184767] LustreError: 188030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779876832 with bad export cookie 7933636864712548382 [ 7231.193391] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7231.205802] LustreError: 188030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7231.491517] Lustre: server umount lustre-MDT0001 complete [ 7240.428116] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 06:14:01 (1779876841) [ 7257.518693] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 7269.027456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7269.372777] LustreError: 192743:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7269.439319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7274.132854] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 7279.587887] LustreError: 192744:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7279.607364] LustreError: 192744:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7283.685096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7289.593193] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 7293.470897] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7301.586554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7301.928644] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7301.941654] Lustre: Skipped 1 previous similar message [ 7307.751238] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 7318.171460] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7318.398954] Lustre: lustre-OST0001: new disk, initializing [ 7318.403647] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7318.421469] Lustre: 195000:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7326.281492] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 7327.900724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7327.926219] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7328.062582] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7337.742190] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7345.480345] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7347.501549] Lustre: DEBUG MARKER: ADD OST3 [ 7352.858258] loop: module loaded [ 7352.952461] loop0: detected capacity change from 0 to 204800000 [ 7353.186272] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7353.206565] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7353.223192] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7359.527482] loop0: detected capacity change from 0 to 204800000 [ 7360.206876] Lustre: DEBUG MARKER: STOP OST3 [ 7366.289602] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7366.520343] Lustre: lustre-OST0003: new disk, initializing [ 7366.530142] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7366.549805] Lustre: 196995:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7366.725163] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7366.737939] Lustre: Skipped 1 previous similar message [ 7368.318560] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7368.530203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7368.543274] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7368.578425] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7370.338288] LustreError: 197146:0:(osp_precreate.c:1281:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7370.573346] Lustre: Failing over lustre-OST0003 [ 7370.779919] Lustre: server umount lustre-OST0003 complete [ 7373.581391] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 7373.727341] LustreError: 194197:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7373.803724] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7373.809347] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7382.744599] Lustre: DEBUG MARKER: START OST3 again [ 7389.315801] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7389.705712] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7390.768409] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7391.725512] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7391.725583] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7395.153918] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 7416.006526] Lustre: Failing over lustre-OST0003 [ 7416.266186] Lustre: server umount lustre-OST0003 complete [ 7417.315920] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7417.321912] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7417.325300] LustreError: 195411:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7417.331784] Lustre: Skipped 2 previous similar messages [ 7417.351049] LustreError: 195411:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 7435.754644] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7435.760871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7435.774141] Lustre: Skipped 3 previous similar messages [ 7435.781114] Lustre: Skipped 2 previous similar messages [ 7442.071837] Lustre: server umount lustre-MDT0000 complete [ 7445.874815] LustreError: 192724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779877047 with bad export cookie 7933636864712549264 [ 7445.891277] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7445.991488] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7445.996018] Lustre: Skipped 2 previous similar messages [ 7460.320300] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7460.509510] Lustre: server umount lustre-MDT0001 complete [ 7464.843314] Lustre: server umount lustre-OST0000 complete [ 7468.829481] Lustre: server umount lustre-OST0001 complete [ 7476.978846] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [ 7485.519863] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 7524.306449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7524.602956] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7524.628578] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7524.721449] Lustre: lustre-MDT0000: new disk, initializing [ 7524.878911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7524.890433] Lustre: Skipped 1 previous similar message [ 7524.910252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7529.192796] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 7540.043049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7540.170201] Lustre: 202384:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7540.491665] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7545.446468] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 7554.281559] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7559.076209] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7566.088322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7566.335982] Lustre: lustre-OST0000: new disk, initializing [ 7566.341839] Lustre: Skipped 1 previous similar message [ 7566.348981] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7566.359623] Lustre: Skipped 2 previous similar messages [ 7566.368806] Lustre: 204125:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7567.934564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7567.944077] Lustre: Skipped 1 previous similar message [ 7567.952051] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7568.113076] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7572.951549] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 7581.247340] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7585.565505] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7585.855072] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7590.001827] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7590.247250] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7593.956609] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7593.960748] LustreError: Skipped 1 previous similar message [ 7593.961962] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7593.965375] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7593.965383] Lustre: Skipped 4 previous similar messages [ 7594.012275] Lustre: Skipped 3 previous similar messages [ 7598.933876] Lustre: server umount lustre-OST0000 complete [ 7610.711322] Lustre: server umount lustre-MDT0000 complete [ 7613.411565] LustreError: 202391:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7613.440980] LustreError: 202391:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 19 previous similar messages [ 7615.196850] LustreError: 202375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779877216 with bad export cookie 7933636864712551560 [ 7615.200942] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7615.213304] LustreError: 202375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7615.829979] Lustre: server umount lustre-MDT0001 complete [ 7619.790865] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 06:20:20 (1779877220) [ 7635.251725] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [ 7647.483501] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 7681.960310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7682.212604] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7682.248874] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7682.353945] Lustre: lustre-MDT0000: new disk, initializing [ 7682.494601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7686.485629] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 7697.714890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7697.860729] Lustre: 208555:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7698.242760] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7702.924632] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 7711.011406] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7715.381228] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7722.951835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7723.421966] Lustre: 210298:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7724.675762] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7730.442770] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 7738.603556] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7742.491206] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7742.690620] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7746.211576] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7746.430798] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7750.116950] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7750.123080] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7750.134556] LustreError: Skipped 1 previous similar message [ 7750.145516] Lustre: Skipped 2 previous similar messages [ 7750.156649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7750.162032] Lustre: Skipped 5 previous similar messages [ 7762.400300] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7762.694602] Lustre: server umount lustre-OST0000 complete [ 7770.083450] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7774.017445] Lustre: server umount lustre-MDT0000 complete [ 7775.719391] LustreError: 208567:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7775.732353] LustreError: 208567:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7777.489559] LustreError: 210297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779877379 with bad export cookie 7933636864712552442 [ 7777.492636] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7777.499566] LustreError: 210297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7777.813410] Lustre: server umount lustre-MDT0001 complete [ 7786.780401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7787.204146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7787.213193] Lustre: Skipped 5 previous similar messages [ 7792.188200] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 7801.995690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7806.918780] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 7814.184808] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7818.570964] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7825.632262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7831.524463] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 7838.107788] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7852.004380] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 7852.010520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7852.015442] Lustre: Skipped 7 previous similar messages [ 7855.861973] Lustre: server umount lustre-OST0000 complete [ 7866.749816] Lustre: server umount lustre-MDT0000 complete [ 7870.817811] LustreError: 212350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779877472 with bad export cookie 7933636864712553324 [ 7870.824598] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7870.829109] LustreError: 212350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7871.142238] Lustre: server umount lustre-MDT0001 complete [ 7979.881405] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [ 7982.897279] Key type lgssc unregistered [ 7983.164407] LNet: 215675:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7983.171304] LNetError: 215675:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7983.200796] LNet: Removed LNI 192.168.204.156@tcp [ 7983.984151] Key type .llcrypt unregistered [ 7983.986585] Key type ._llcrypt unregistered [ 8011.129407] Key type ._llcrypt registered [ 8011.134382] Key type .llcrypt registered [ 8011.255211] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [ 8028.503803] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 8029.601531] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8029.622069] alg: No test for adler32 (adler32-zlib) [ 8030.788631] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 8031.083395] LNet: Added LNI 192.168.204.156@tcp [8/256/0/180] [ 8032.792251] Key type lgssc registered [ 8034.362208] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8078.902405] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8078.933315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8080.313833] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8080.355690] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8080.486788] Lustre: lustre-MDT0000: new disk, initializing [ 8080.605674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8080.631152] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8085.474759] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 8098.163955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8098.245659] Lustre: 218997:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8098.293523] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8098.296489] Lustre: Skipped 1 previous similar message [ 8098.379593] Lustre: lustre-MDT0001: new disk, initializing [ 8098.536515] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8098.591506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8098.612351] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8103.301218] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 8112.836773] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8118.050842] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8127.391501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8127.889984] Lustre: lustre-OST0000: new disk, initializing [ 8127.900831] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8127.916393] Lustre: 220742:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8128.086690] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8134.736044] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 8135.720424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8135.724324] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8135.837533] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8143.345629] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8149.599648] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8149.943086] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8154.604748] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8154.842244] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8161.257660] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8161.270475] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8161.289624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8161.778879] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8161.779845] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8161.804161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8163.139290] Lustre: server umount lustre-OST0000 complete [ 8170.468873] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8170.482174] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8170.501617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8177.121821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8177.128869] Lustre: Skipped 3 previous similar messages [ 8181.216166] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8181.545482] Lustre: server umount lustre-MDT0000 complete [ 8186.039707] LustreError: 218988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779877787 with bad export cookie 1240746816861183927 [ 8186.045857] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8186.057557] LustreError: 218988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8186.551574] Lustre: server umount lustre-MDT0001 complete [ 8211.938626] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 8224.180314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8224.711431] LustreError: 223533:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8224.727660] LustreError: 223533:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8224.779187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8229.876114] LustreError: 223534:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8230.046814] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 8234.999669] LustreError: 223533:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8239.139445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8239.619852] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8244.961181] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 8248.817148] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8257.681965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8258.107845] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8265.494192] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 8275.676994] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8275.904184] Lustre: lustre-OST0001: new disk, initializing [ 8275.907961] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8275.916610] Lustre: 225789:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8276.022627] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8283.242520] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8283.251881] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8283.414312] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8283.752462] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 8295.333988] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8303.669636] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8320.179799] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 06:32:00 (1779877920) [ 8324.585042] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8324.593455] Lustre: Skipped 2 previous similar messages [ 8324.604021] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8324.613581] Lustre: Skipped 3 previous similar messages [ 8328.944299] Lustre: server umount lustre-MDT0000 complete [ 8329.701422] LustreError: 223534:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8329.718946] LustreError: 223534:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8333.300785] LustreError: 223515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779877934 with bad export cookie 1240746816861184809 [ 8333.306128] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8333.313519] LustreError: 223515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8333.646167] Lustre: server umount lustre-MDT0001 complete [ 8348.561115] Lustre: server umount lustre-OST0000 complete [ 8351.200142] Lustre: 217221:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779877936/real 1779877936] req@ffff8cca8c0fdf80 x1866336967967104/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779877952 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8351.241125] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8351.254979] Lustre: Skipped 1 previous similar message [ 8352.851977] Lustre: server umount lustre-OST0001 complete [ 8360.481387] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [ 8375.803856] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 8420.436646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8420.830459] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8420.890930] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8421.026560] Lustre: lustre-MDT0000: new disk, initializing [ 8421.139946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8421.158777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8426.433510] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 8437.109892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8437.259760] Lustre: 229708:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8437.310081] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8437.312594] Lustre: Skipped 1 previous similar message [ 8437.378247] Lustre: lustre-MDT0001: new disk, initializing [ 8437.477238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8437.483996] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8442.517476] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 8451.448567] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8455.542945] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8463.289713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8463.736530] Lustre: lustre-OST0000: new disk, initializing [ 8463.741870] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8463.750269] Lustre: 231452:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8465.431562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8465.443073] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8465.564527] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8471.303887] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 8478.973903] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8485.102898] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8485.477353] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8492.227815] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8492.518104] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8496.106328] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8496.128916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8496.134795] Lustre: Skipped 1 previous similar message [ 8501.908960] Lustre: server umount lustre-OST0000 complete [ 8512.617874] Lustre: server umount lustre-MDT0000 complete [ 8516.247995] LustreError: 229700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779878117 with bad export cookie 1240746816861186944 [ 8516.251611] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8516.255856] LustreError: 229700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8516.504767] Lustre: server umount lustre-MDT0001 complete [ 8525.667505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8526.037953] LustreError: 233520:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8526.122925] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8526.132027] Lustre: Skipped 2 previous similar messages [ 8531.076972] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 8535.522711] LustreError: 233521:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8535.561130] LustreError: 233521:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8540.585357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8545.557291] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 8552.407733] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8556.750805] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8565.502774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8565.733535] LustreError: 235115:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8565.765900] LustreError: 235115:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8572.646920] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 8579.969464] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8728.033922] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8728.037055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8728.050183] Lustre: Skipped 4 previous similar messages [ 8728.062786] Lustre: Skipped 7 previous similar messages [ 8733.645710] Lustre: server umount lustre-OST0000 complete [ 8752.103349] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8752.337813] Lustre: server umount lustre-MDT0000 complete [ 8753.633352] LustreError: 233521:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8753.676530] LustreError: 233521:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8756.713543] LustreError: 233503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779878358 with bad export cookie 1240746816861187826 [ 8756.720240] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8756.736652] LustreError: 233503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8757.111201] Lustre: server umount lustre-MDT0001 complete [ 8847.687513] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [ 8850.504840] Key type lgssc unregistered [ 8850.799594] LNet: 236771:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8850.805986] LNetError: 236771:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8850.823837] LNet: Removed LNI 192.168.204.156@tcp [ 8851.781107] Key type .llcrypt unregistered [ 8851.784045] Key type ._llcrypt unregistered [ 8880.366276] Key type ._llcrypt registered [ 8880.368650] Key type .llcrypt registered [ 8880.506494] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [ 8897.504839] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 8899.079149] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8899.161045] alg: No test for adler32 (adler32-zlib) [ 8900.310775] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [ 8900.506767] LNet: Added LNI 192.168.204.156@tcp [8/256/0/180] [ 8902.192422] Key type lgssc registered [ 8903.450890] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8956.878192] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 8966.685703] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8966.705181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8967.962684] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8967.994240] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8968.091643] Lustre: lustre-MDT0000: new disk, initializing [ 8968.191789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8968.214497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8972.822101] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 8983.926945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8984.076100] Lustre: 240794:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8984.168990] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8984.173092] Lustre: Skipped 1 previous similar message [ 8984.258415] Lustre: lustre-MDT0001: new disk, initializing [ 8984.336237] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8984.372837] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8984.398965] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8988.789797] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 8993.774962] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9000.022571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9000.257949] Lustre: lustre-OST0000: new disk, initializing [ 9000.266233] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9000.281458] Lustre: 242389:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9000.357813] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9005.921419] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 9006.181650] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9006.213781] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9006.362915] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9017.239282] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9017.439213] Lustre: lustre-OST0001: new disk, initializing [ 9017.443595] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9017.449437] Lustre: 243246:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9017.552772] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9023.533735] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 9025.603195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9025.611934] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9025.695643] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9034.872412] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9041.105697] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9051.624115] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9051.629202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9051.642111] Lustre: Skipped 1 previous similar message [ 9051.651477] Lustre: Skipped 2 previous similar messages [ 9056.225635] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9056.231777] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9056.244791] Lustre: Skipped 1 previous similar message [ 9056.254468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9057.802672] Lustre: server umount lustre-MDT0000 complete [ 9061.677382] LustreError: 242399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779878663 with bad export cookie 15680471401770502467 [ 9061.678819] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9061.698433] LustreError: 242399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9061.858274] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9061.864447] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9061.873305] Lustre: Skipped 1 previous similar message [ 9061.886741] Lustre: Skipped 4 previous similar messages [ 9062.099158] Lustre: server umount lustre-MDT0001 complete [ 9068.747875] Lustre: server umount lustre-OST0000 complete [ 9073.244504] Lustre: server umount lustre-OST0001 complete [ 9090.680415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9100.753830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9110.195564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9119.604542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9141.275145] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 9151.544426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9151.586820] Lustre: lustre-MDT0000: reset Object Index mappings [ 9152.066346] LustreError: 248222:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9152.080389] LustreError: 248222:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9152.171576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9156.539453] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 9157.604752] LustreError: 248223:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9161.699955] LustreError: 248222:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9166.282919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9166.322811] Lustre: lustre-MDT0001: reset Object Index mappings [ 9166.802962] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9171.303249] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 9174.764750] Lustre: 249330:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9182.324656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9189.633988] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 9192.046990] LustreError: 249684:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9197.181228] LustreError: 250146:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9197.202443] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9197.208591] LustreError: 250146:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9199.109585] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9199.440946] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9199.450131] Lustre: Skipped 1 previous similar message [ 9204.746462] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9205.721674] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 9216.164286] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9221.165850] Lustre: 251168:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9230.309926] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9230.315214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9230.330034] Lustre: Skipped 1 previous similar message [ 9230.345966] Lustre: Skipped 3 previous similar messages [ 9234.614717] Lustre: server umount lustre-MDT0000 complete [ 9235.441757] LustreError: 248218:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9235.469123] LustreError: 248218:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9239.021492] LustreError: 248204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779878840 with bad export cookie 15680471401770509257 [ 9239.034144] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9239.447699] Lustre: server umount lustre-MDT0001 complete [ 9253.569085] Lustre: server umount lustre-OST0000 complete [ 9256.546439] Lustre: 238318:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779878842/real 1779878842] req@ffff8ccbb6745500 x1866337880240896/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779878858 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9256.561759] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9256.574472] Lustre: Skipped 2 previous similar messages [ 9258.609547] Lustre: server umount lustre-OST0001 complete [ 9281.354545] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 06:48:01 (1779878881) [ 9300.253080] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [ 9312.335471] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 9353.321937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9353.595392] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9353.635704] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9353.746516] Lustre: lustre-MDT0000: new disk, initializing [ 9353.858463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9353.877486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9358.162334] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 9369.207161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9369.341673] Lustre: 255392:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9369.406245] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9369.411577] Lustre: Skipped 1 previous similar message [ 9369.491246] Lustre: lustre-MDT0001: new disk, initializing [ 9369.644326] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9369.660332] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9374.018767] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 9381.612920] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9386.473296] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9393.710792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9393.930601] Lustre: lustre-OST0000: new disk, initializing [ 9393.934760] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9393.941338] Lustre: 257133:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9395.149604] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9395.176461] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9395.297467] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9400.274453] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 9408.188911] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9414.176193] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9414.344767] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9420.174488] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9420.503258] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9425.891845] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9425.893734] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9425.897530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9425.904897] Lustre: Skipped 1 previous similar message [ 9429.280160] Lustre: server umount lustre-OST0000 complete [ 9436.131173] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9436.136930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9436.161898] Lustre: Skipped 2 previous similar messages [ 9447.392332] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9447.604661] Lustre: server umount lustre-MDT0000 complete [ 9451.485847] LustreError: 255384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779879053 with bad export cookie 15680471401770511350 [ 9451.487288] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9451.496602] LustreError: 255384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 9451.814666] Lustre: server umount lustre-MDT0001 complete [ 9473.315755] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 9485.757401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9486.250397] LustreError: 259905:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9486.329438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9486.339700] Lustre: Skipped 2 previous similar messages [ 9490.834655] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 9500.835135] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9506.033577] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 9509.188326] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9515.935173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9522.357536] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 9531.418641] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9531.567971] Lustre: lustre-OST0001: new disk, initializing [ 9531.577342] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9531.592368] Lustre: 262162:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9533.520472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9533.536460] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9533.650398] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9538.188463] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 9548.670856] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9552.893464] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9559.271344] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 9559.393318] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 9567.716432] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9567.733475] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9567.747033] Lustre: Skipped 2 previous similar messages [ 9567.755780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9567.764136] Lustre: Skipped 4 previous similar messages [ 9573.098718] Lustre: server umount lustre-MDT0000 complete [ 9574.894888] LustreError: 259900:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9574.923487] LustreError: 259900:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 9577.090744] LustreError: 261378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779879178 with bad export cookie 15680471401770512232 [ 9577.093433] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9577.097684] LustreError: 261378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9577.390491] Lustre: server umount lustre-MDT0001 complete [ 9592.094891] Lustre: server umount lustre-OST0000 complete [ 9606.550333] Lustre: server umount lustre-OST0001 complete [ 9625.200830] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 9635.829595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9636.500418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9636.503539] Lustre: Skipped 3 previous similar messages [ 9641.463664] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 9641.953496] LustreError: 265188:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9641.975746] LustreError: 265188:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9649.911324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9655.400879] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 9659.679704] Lustre: 266295:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9669.771640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9677.856879] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 9681.927328] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 9687.784704] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9693.177862] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 9695.268512] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 9703.960640] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9708.650092] Lustre: 268139:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9714.451055] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 9714.556577] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 9726.571335] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 06:55:26 (1779879326) [ 9742.306154] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9742.313684] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9742.324143] Lustre: Skipped 3 previous similar messages [ 9742.330876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9742.340953] Lustre: Skipped 4 previous similar messages [ 9748.128343] Lustre: server umount lustre-MDT0000 complete [ 9752.124744] LustreError: 265169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779879353 with bad export cookie 15680471401770518553 [ 9752.135788] LustreError: 265169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9752.138135] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9758.498620] Lustre: server umount lustre-MDT0001 complete [ 9768.594221] Lustre: server umount lustre-OST0000 complete [ 9773.171592] Lustre: server umount lustre-OST0001 complete [ 9781.291833] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [ 9793.306164] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 9845.908274] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [ 9855.740368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9855.977402] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9856.011089] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9856.111851] Lustre: lustre-MDT0000: new disk, initializing [ 9856.201326] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9860.075081] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 9869.583118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9869.674916] Lustre: 272331:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9869.887278] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9873.806434] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 9884.839929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9885.090473] Lustre: 273930:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9886.452472] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9890.638715] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 9902.185974] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9902.346541] Lustre: 274785:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9902.487243] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9902.494651] Lustre: Skipped 6 previous similar messages [ 9904.105907] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9904.111507] Lustre: Skipped 1 previous similar message [ 9904.254029] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9910.102768] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 9920.511165] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9924.409433] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9924.414825] Lustre: Skipped 1 previous similar message [ 9940.219877] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 06:59:00 (1779879540) [ 9944.380889] Lustre: server umount lustre-MDT0000 complete [ 9945.059464] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9945.063724] LustreError: 272338:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9945.075232] Lustre: Skipped 6 previous similar messages [ 9945.094805] LustreError: 272338:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 18 previous similar messages [ 9948.510640] LustreError: 272323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779879550 with bad export cookie 15680471401770526428 [ 9948.515294] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9948.525902] LustreError: 272323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9948.814234] Lustre: server umount lustre-MDT0001 complete [ 9962.534646] Lustre: server umount lustre-OST0000 complete [ 9965.537268] Lustre: 238316:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779879551/real 1779879551] req@ffff8ccba622a300 x1866337880501632/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1779879567 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9965.932613] Lustre: server umount lustre-OST0001 complete [ 9974.375542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9979.713752] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 9988.346535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9992.758906] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [ 9998.417754] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10002.660049] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10009.585813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10015.577337] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [10021.205599] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10038.179403] loop1: detected capacity change from 0 to 204800000 [10038.314302] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10038.330862] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10038.345655] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10043.407167] loop1: detected capacity change from 0 to 204800000 [10049.598892] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10049.687392] LustreError: 277003:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) fs24271: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10049.707138] LustreError: 277003:0:(mgs_llog.c:731:mgs_check_index()) Can't get db for fs24271 [10049.712232] LustreError: 277003:0:(mgs_handler.c:517:mgs_target_reg()) fs24271-MDT0001: Error checking target index: rc = -13 [10049.723961] LustreError: fs24271-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10049.737715] LustreError: 280440:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -13 [10049.745669] LustreError: 280440:0:(tgt_mount.c:2084:server_put_super()) no obd fs24271-MDT0001 [10049.763555] LustreError: 280440:0:(tgt_mount.c:116:server_deregister_mount()) fs24271-MDT0001 not registered [10049.807038] Lustre: server umount fs24271-MDT0001 complete [10049.812353] LustreError: 280440:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [10059.086651] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10059.157579] Lustre: Setting parameter fs24271-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs24271-MDT0001 [10059.194740] Lustre: srv-fs24271-MDT0001: No data found on store. Initialize space. [10059.198241] Lustre: Skipped 4 previous similar messages [10059.307649] Lustre: fs24271-MDT0001: new disk, initializing [10059.327940] Lustre: Skipped 3 previous similar messages [10059.589595] Lustre: srv-fs24271-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10059.602706] Lustre: 280906:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs24271-MDT0001-mdtlov: can't create new llog: rc = -115 [10059.621444] LustreError: 280906:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs24271-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10063.907262] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [10071.075756] Lustre: server umount fs24271-MDT0001 complete [10083.856790] loop1: detected capacity change from 0 to 204800000 [10089.468350] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10089.965288] Lustre: 282469:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs24271-MDT0001-mdtlov: can't create new llog: rc = -115 [10089.978150] LustreError: 282469:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs24271-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10094.760650] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [10097.140377] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10099.907787] Lustre: server umount fs24271-MDT0001 complete [10111.462602] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10111.475437] Lustre: Skipped 3 previous similar messages [10111.480287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10111.486656] Lustre: Skipped 8 previous similar messages [10116.310079] Lustre: server umount lustre-MDT0000 complete [10124.783793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10124.903947] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10129.913868] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [10130.415870] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10137.855927] loop1: detected capacity change from 0 to 204800000 [10142.935250] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10143.436924] Lustre: 284880:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs24271-MDT0001-mdtlov: can't create new llog: rc = -115 [10143.446639] LustreError: 284880:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs24271-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10148.298641] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [10153.224808] Lustre: server umount fs24271-MDT0001 complete [10176.498021] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10176.512784] LustreError: Skipped 2 previous similar messages [10181.543879] Lustre: server umount lustre-OST0000 complete [10191.902068] Lustre: server umount lustre-MDT0000 complete [10195.598770] LustreError: 276994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779879797 with bad export cookie 15680471401770529109 [10195.603266] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10195.606113] LustreError: 276994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10195.990106] Lustre: server umount lustre-MDT0001 complete [10207.205345] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [10210.222461] Key type lgssc unregistered [10210.555989] LNet: 287192:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10210.563057] LNetError: 287192:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10210.587565] LNet: Removed LNI 192.168.204.156@tcp [10211.398233] Key type .llcrypt unregistered [10211.403538] Key type ._llcrypt unregistered [10231.781148] Key type ._llcrypt registered [10231.782790] Key type .llcrypt registered [10231.919813] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [10233.060818] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10233.207651] alg: No test for adler32 (adler32-zlib) [10234.434424] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [10234.748983] LNet: Added LNI 192.168.204.156@tcp [8/256/0/180] [10236.452343] Key type lgssc registered [10237.708025] Lustre: Echo OBD driver; http://www.lustre.org/ [10247.336391] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10247.368207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10248.904228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10254.382617] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [10258.408245] LustreError: 288449:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10258.424558] LustreError: 288449:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10262.999427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10263.490257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10267.745298] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [10273.258634] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10277.254807] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10284.847529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10285.158862] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10288.246400] LustreError: 290050:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10290.810225] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [10293.746980] LustreError: 290052:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10295.425718] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10297.873886] LustreError: 290052:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10297.904456] LustreError: 290052:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10303.344791] LustreError: 290235:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10303.363841] LustreError: 290235:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10313.304789] loop1: detected capacity change from 0 to 204800000 [10313.433883] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10313.443909] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10313.458569] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10313.581710] LustreError: 290050:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10313.598859] LustreError: 290050:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [10318.054339] loop1: detected capacity change from 0 to 204800000 [10322.993445] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10323.126936] Lustre: fs3914-OST0000: new disk, initializing [10323.134543] Lustre: srv-fs3914-OST0000: No data found on store. Initialize space. [10323.148513] Lustre: 291892:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10323.217507] Lustre: fs3914-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10329.315990] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [10330.602913] LustreError: 290053:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10330.620842] LustreError: 290053:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [10337.619807] Lustre: server umount fs3914-OST0000 complete [10348.188648] loop1: detected capacity change from 0 to 204800000 [10353.164825] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10353.371979] Lustre: fs3914-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10359.221762] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [10364.794550] LustreError: 290053:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10364.814608] LustreError: 290053:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 18 previous similar messages [10367.832967] loop2: detected capacity change from 0 to 204800000 [10368.003145] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10368.019147] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10368.033474] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10371.560618] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10372.991917] loop2: detected capacity change from 0 to 204800000 [10376.675368] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10378.272301] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10378.383934] LustreError: fs3914-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10378.398267] LustreError: fs3914-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10378.415688] LustreError: 294900:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -13 [10378.425918] LustreError: 294900:0:(tgt_mount.c:2084:server_put_super()) no obd fs3914-OST0001 [10378.432980] LustreError: 294900:0:(tgt_mount.c:116:server_deregister_mount()) fs3914-OST0001 not registered [10378.478780] Lustre: server umount fs3914-OST0001 complete [10378.482457] LustreError: 294900:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [10389.272359] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10389.454676] Lustre: fs3914-OST0001: new disk, initializing [10389.460488] Lustre: srv-fs3914-OST0001: No data found on store. Initialize space. [10389.469940] Lustre: 295451:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10389.549890] Lustre: fs3914-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10396.754475] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [10405.847712] Lustre: server umount fs3914-OST0000 complete [10414.846596] Lustre: server umount fs3914-OST0001 complete [10428.672349] loop1: detected capacity change from 0 to 204800000 [10431.363066] LustreError: 290235:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10431.399511] LustreError: 290235:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 38 previous similar messages [10434.370202] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10434.653101] Lustre: fs3914-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10440.087359] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [10446.834279] loop2: detected capacity change from 0 to 204800000 [10452.506648] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10458.982300] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [10467.611864] Lustre: server umount fs3914-OST0000 complete [10476.394984] Lustre: server umount fs3914-OST0001 complete [10488.292912] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10488.303675] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10488.323911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10489.314804] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10489.316756] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10494.440274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10494.446767] Lustre: Skipped 2 previous similar messages [10499.561532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10499.569471] Lustre: Skipped 1 previous similar message [10502.112201] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10502.270255] Lustre: server umount lustre-OST0000 complete [10509.287166] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10509.298115] LustreError: Skipped 1 previous similar message [10509.304215] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10509.322208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10512.198471] Lustre: server umount lustre-MDT0000 complete [10515.721980] LustreError: 295450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779880117 with bad export cookie 3562084285968109014 [10515.726169] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10515.730269] LustreError: 295450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10516.179639] Lustre: server umount lustre-MDT0001 complete [10526.764733] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [10529.761957] Key type lgssc unregistered [10529.996659] LNet: 301307:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10530.005926] LNetError: 301307:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10530.017061] LNet: Removed LNI 192.168.204.156@tcp [10530.886165] Key type .llcrypt unregistered [10530.895049] Key type ._llcrypt unregistered [10561.350660] Key type ._llcrypt registered [10561.352733] Key type .llcrypt registered [10561.438354] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [10575.639483] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [10576.858965] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10576.942505] alg: No test for adler32 (adler32-zlib) [10577.929065] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [10578.115566] LNet: Added LNI 192.168.204.156@tcp [8/256/0/180] [10579.785828] Key type lgssc registered [10580.827795] Lustre: Echo OBD driver; http://www.lustre.org/ [10618.640232] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10618.670847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10619.957599] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10620.019548] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10620.104589] Lustre: lustre-MDT0000: new disk, initializing [10620.233261] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10620.252963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10624.470599] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [10635.816498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10635.966149] Lustre: 305131:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10636.096616] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10636.106860] Lustre: Skipped 1 previous similar message [10636.222333] Lustre: lustre-MDT0001: new disk, initializing [10636.355375] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10636.399145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10636.411278] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10641.719857] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [10649.867711] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10653.527476] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10660.428682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10660.751384] Lustre: lustre-OST0000: new disk, initializing [10660.757453] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10660.766539] Lustre: 306874:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10660.848429] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10667.040748] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [10669.639894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10669.652033] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10669.760180] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10674.374371] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10676.415324] Lustre: Mounted lustre-client [10677.238922] LustreError: 307624:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff8cca82188000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [10677.343176] Lustre: Unmounted lustre-client [10686.955950] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10686.959060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10686.968438] Lustre: Skipped 1 previous similar message [10686.978879] Lustre: Skipped 1 previous similar message [10692.067578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10692.077622] Lustre: Skipped 1 previous similar message [10697.191815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10697.198568] Lustre: Skipped 1 previous similar message [10700.256113] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10700.475111] Lustre: server umount lustre-OST0000 complete [10707.433307] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10707.461251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10707.472725] Lustre: Skipped 1 previous similar message [10710.206126] Lustre: server umount lustre-MDT0000 complete [10714.017079] LustreError: 305124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779880315 with bad export cookie 11764838515020105076 [10714.026515] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10714.028047] LustreError: 305124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10714.457252] Lustre: server umount lustre-MDT0001 complete [10725.886366] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [10729.259513] Key type lgssc unregistered [10734.880192] LNet: 1 local NIs in recovery (showing 1): 192.168.204.156@tcp [10735.712783] LNet: 308907:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10735.741515] LNetError: 308907:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10806.451225] LNet: Removed LNI 192.168.204.156@tcp [10807.228557] Key type .llcrypt unregistered [10807.231144] Key type ._llcrypt unregistered [10824.943471] Key type ._llcrypt registered [10824.944994] Key type .llcrypt registered [10825.034243] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [10839.768587] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [10841.028585] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10841.117566] alg: No test for adler32 (adler32-zlib) [10842.235929] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [10842.588326] LNet: Added LNI 192.168.204.156@tcp [8/256/0/180] [10844.304256] Key type lgssc registered [10845.554659] Lustre: Echo OBD driver; http://www.lustre.org/ [10879.559980] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 07:14:39 (1779880479) [10887.181697] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10887.244235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10888.574793] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10888.604912] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10888.679395] Lustre: lustre-MDT0000: new disk, initializing [10888.817604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10888.911615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10892.902949] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [10904.601446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10904.750583] Lustre: 312096:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10904.840422] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10904.852400] Lustre: Skipped 1 previous similar message [10905.066071] Lustre: lustre-MDT0001: new disk, initializing [10905.205217] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10905.253334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10905.274633] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10910.092860] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [10919.388956] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10924.278457] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10931.392404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10931.862960] Lustre: lustre-OST0000: new disk, initializing [10931.869810] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10931.881258] Lustre: 313840:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10931.972365] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10937.954489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10937.983326] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10938.160279] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10939.083204] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [10948.457417] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10963.938639] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10963.942928] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10963.953806] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10963.990123] Lustre: Skipped 1 previous similar message [10965.783342] Lustre: server umount lustre-OST0000 complete [10971.627376] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10971.635470] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10971.657179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10971.660171] Lustre: Skipped 1 previous similar message [10974.178238] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10974.179195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10974.219355] Lustre: Skipped 1 previous similar message [10975.966650] Lustre: server umount lustre-MDT0000 complete [10979.632157] LustreError: 312089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779880581 with bad export cookie 17040494174297695005 [10979.640794] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10979.642312] LustreError: 312089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10980.000393] Lustre: server umount lustre-MDT0001 complete [10990.493340] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [10992.944537] Key type lgssc unregistered [10993.236692] LNet: 315471:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10993.246678] LNetError: 315471:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10993.275631] LNet: Removed LNI 192.168.204.156@tcp [10994.074848] Key type .llcrypt unregistered [10994.079326] Key type ._llcrypt unregistered [11018.712369] Key type ._llcrypt registered [11018.714478] Key type .llcrypt registered [11018.818877] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [11019.854175] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11019.971457] alg: No test for adler32 (adler32-zlib) [11021.051758] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [11021.230441] LNet: Added LNI 192.168.204.156@tcp [8/256/0/180] [11022.904318] Key type lgssc registered [11024.048844] Lustre: Echo OBD driver; http://www.lustre.org/ [11031.444343] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11031.471968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11032.982436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11036.849272] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [11043.300395] LustreError: 316875:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11043.318123] LustreError: 316875:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11045.435932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11045.722957] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11050.058847] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [11056.490630] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11060.385960] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11066.790232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11067.110047] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11073.291836] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [11079.230262] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11086.438486] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11088.158083] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11091.937561] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11091.956053] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11091.973129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11093.986546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11093.997649] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11094.010035] Lustre: Skipped 1 previous similar message [11096.470251] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.56@tcp (stopping) [11096.483034] Lustre: Skipped 2 previous similar messages [11097.971612] Lustre: server umount lustre-MDT0000 complete [11099.114112] LustreError: 316875:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11099.144151] LustreError: 316875:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11101.548603] LustreError: 316871:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11102.082098] LustreError: 316856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779880703 with bad export cookie 4739815047806599740 [11102.098426] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11102.101122] LustreError: 316856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11104.225293] LustreError: 316872:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11104.232773] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11104.279476] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11108.525847] Lustre: server umount lustre-MDT0001 complete [11112.454846] Lustre: server umount lustre-OST0000 complete [11121.432130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11125.975119] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [11128.357680] Lustre: 320602:0:(mgs_llog.c:1798:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.156@tcp [11130.425881] Lustre: server umount MGS complete [11138.942192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11139.542648] LustreError: 321173:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11139.568378] LustreError: 321173:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11139.624573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11144.346492] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [11149.795139] LustreError: 321174:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11149.812804] LustreError: 321174:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11152.755292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11153.093467] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11157.435651] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [11166.719276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11167.002731] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11173.492628] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [11179.461343] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11181.465925] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11186.016611] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11187.647877] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11193.825431] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11193.834092] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11193.849196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11193.854477] Lustre: Skipped 2 previous similar messages [11194.866786] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11199.538868] Lustre: server umount lustre-OST0000 complete [11204.578109] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11204.599984] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11204.623865] Lustre: Skipped 1 previous similar message [11204.629256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11204.640972] Lustre: Skipped 2 previous similar messages [11210.575186] Lustre: server umount lustre-MDT0000 complete [11214.366587] LustreError: 321154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779880816 with bad export cookie 4739815047806600580 [11214.369382] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11214.375660] LustreError: 321154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [11214.722596] Lustre: server umount lustre-MDT0001 complete [11225.709958] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [11228.567712] Key type lgssc unregistered [11228.889422] LNet: 324350:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11228.898520] LNetError: 324350:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11228.920483] LNet: Removed LNI 192.168.204.156@tcp [11229.785764] Key type .llcrypt unregistered [11229.788211] Key type ._llcrypt unregistered [11246.791906] Key type ._llcrypt registered [11246.798054] Key type .llcrypt registered [11246.939582] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [11262.729724] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [11264.491388] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11264.885157] alg: No test for adler32 (adler32-zlib) [11266.023698] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [11266.234756] LNet: Added LNI 192.168.204.156@tcp [8/256/0/180] [11267.904450] Key type lgssc registered [11269.635762] Lustre: Echo OBD driver; http://www.lustre.org/ [11305.053373] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11305.078462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11306.397617] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11306.422941] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11306.494325] Lustre: lustre-MDT0000: new disk, initializing [11306.589849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11306.611102] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11310.809398] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [11321.227046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11321.409509] Lustre: 327477:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11321.502825] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11321.506744] Lustre: Skipped 1 previous similar message [11321.617781] Lustre: lustre-MDT0001: new disk, initializing [11321.755910] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11321.823371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11321.837431] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11327.303467] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [11336.089508] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11340.177552] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11347.814391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11348.093937] Lustre: lustre-OST0000: new disk, initializing [11348.097457] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11348.102961] Lustre: 329217:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11348.185783] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11353.515204] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [11357.799912] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11357.815299] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11357.943087] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11361.204292] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11366.143377] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11366.429388] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11371.118417] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11371.305158] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11373.540769] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11373.541517] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11373.551396] LustreError: Skipped 1 previous similar message [11373.553839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11373.559369] Lustre: Skipped 1 previous similar message [11378.670095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11378.686549] Lustre: Skipped 2 previous similar messages [11379.443339] Lustre: server umount lustre-OST0000 complete [11383.265762] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11383.272088] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11383.291417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11388.897742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11388.911943] Lustre: Skipped 2 previous similar messages [11389.325513] Lustre: server umount lustre-MDT0000 complete [11393.004531] LustreError: 327467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779880994 with bad export cookie 6017826837721096002 [11393.007186] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11393.016740] LustreError: 327467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11393.488587] Lustre: server umount lustre-MDT0001 complete [11402.765254] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 07:23:23 (1779881003) [11420.581986] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [11430.883451] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [11470.727291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11471.030097] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11471.065923] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11471.140679] Lustre: lustre-MDT0000: new disk, initializing [11471.218868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11471.233308] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11476.177252] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [11486.862636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11486.965481] Lustre: 333796:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11487.048965] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11487.055591] Lustre: Skipped 1 previous similar message [11487.146150] Lustre: lustre-MDT0001: new disk, initializing [11487.291421] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11487.352666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11487.369769] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11491.578961] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [11499.315292] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11503.477532] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11510.395641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11510.644858] Lustre: lustre-OST0000: new disk, initializing [11510.648719] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11510.653950] Lustre: 335540:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11510.748727] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11516.585301] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [11517.517112] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11517.529238] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11517.631056] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11523.714763] Lustre: DEBUG MARKER: oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11558.884030] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11558.885151] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11558.886171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11558.898844] LustreError: Skipped 1 previous similar message [11558.915850] Lustre: Skipped 2 previous similar messages [11562.569969] Lustre: server umount lustre-OST0000 complete [11568.049064] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11568.052850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11568.077810] Lustre: Skipped 1 previous similar message [11568.083208] Lustre: Skipped 1 previous similar message [11573.019919] Lustre: server umount lustre-MDT0000 complete [11577.125590] LustreError: 333788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779881178 with bad export cookie 6017826837721096884 [11577.127826] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11577.135328] LustreError: 333788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11577.451846] Lustre: server umount lustre-MDT0001 complete [11589.702431] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [11594.832236] Key type lgssc unregistered [11595.555312] LNet: 337586:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11595.563775] LNetError: 337586:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11595.592547] LNet: Removed LNI 192.168.204.156@tcp [11597.141965] Key type .llcrypt unregistered [11597.144616] Key type ._llcrypt unregistered [11688.611975] Key type ._llcrypt registered [11688.618971] Key type .llcrypt registered [11688.735191] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [11689.054707] Key type .llcrypt unregistered [11689.059716] Key type ._llcrypt unregistered [11765.485625] Key type ._llcrypt registered [11765.492665] Key type .llcrypt registered [11765.623957] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [11765.897442] Key type .llcrypt unregistered [11765.899580] Key type ._llcrypt unregistered [11798.724204] Key type ._llcrypt registered [11798.733466] Key type .llcrypt registered [11798.843809] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [11799.163497] Key type .llcrypt unregistered [11799.167927] Key type ._llcrypt unregistered [11830.935734] Key type ._llcrypt registered [11830.938716] Key type .llcrypt registered [11831.027444] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [11831.271286] Key type .llcrypt unregistered [11831.273090] Key type ._llcrypt unregistered [11915.850717] Key type ._llcrypt registered [11915.852314] Key type .llcrypt registered [11915.949219] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [11916.256318] Key type .llcrypt unregistered [11916.265385] Key type ._llcrypt unregistered [11944.858412] Key type ._llcrypt registered [11944.865633] Key type .llcrypt registered [11944.983916] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [11945.302345] Key type .llcrypt unregistered [11945.304056] Key type ._llcrypt unregistered [11980.130221] Key type ._llcrypt registered [11980.134871] Key type .llcrypt registered [11980.256758] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [11980.629647] Key type .llcrypt unregistered [11980.631284] Key type ._llcrypt unregistered [12012.270519] Key type ._llcrypt registered [12012.272202] Key type .llcrypt registered [12012.364936] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [12012.675371] Key type .llcrypt unregistered [12012.678888] Key type ._llcrypt unregistered [12045.142449] Key type ._llcrypt registered [12045.144987] Key type .llcrypt registered [12045.267891] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [12045.586455] Key type .llcrypt unregistered [12045.588831] Key type ._llcrypt unregistered [12077.705331] Key type ._llcrypt registered [12077.708451] Key type .llcrypt registered [12077.830637] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [12078.082212] Key type .llcrypt unregistered [12078.084706] Key type ._llcrypt unregistered [12138.536308] Key type ._llcrypt registered [12138.537904] Key type .llcrypt registered [12138.666786] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [12138.949280] Key type .llcrypt unregistered [12138.953261] Key type ._llcrypt unregistered [12169.534496] Key type ._llcrypt registered [12169.537498] Key type .llcrypt registered [12169.663429] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [12170.066556] Key type .llcrypt unregistered [12170.070565] Key type ._llcrypt unregistered [12204.333978] Key type ._llcrypt registered [12204.335310] Key type .llcrypt registered [12204.432266] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [12204.670144] Key type .llcrypt unregistered [12204.673805] Key type ._llcrypt unregistered [12261.464957] Key type ._llcrypt registered [12261.466923] Key type .llcrypt registered [12261.678797] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [12280.726532] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [12281.925491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12282.077361] alg: No test for adler32 (adler32-zlib) [12283.252580] Lustre: Lustre: Build Version: 2.17.53_24_g30bdaff [12283.495332] LNet: Added LNI 192.168.204.156@tcp [8/256/0/180] [12285.216164] Key type lgssc registered [12286.477578] Lustre: Echo OBD driver; http://www.lustre.org/ [12327.478482] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12327.511591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12328.816424] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [12328.840283] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [12328.902255] Lustre: tst250fs-MDT0000: new disk, initializing [12328.983185] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12329.003661] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12334.006958] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [12346.406655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12346.558405] Lustre: 357634:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12346.668282] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [12346.682559] Lustre: Skipped 1 previous similar message [12346.858664] Lustre: tst250fs-MDT0001: new disk, initializing [12346.962911] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12347.008076] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12347.020903] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12351.774765] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [12357.016508] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12364.255613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12364.599861] Lustre: tst250fs-OST0000: new disk, initializing [12364.604364] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [12364.618317] Lustre: 359233:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12364.762179] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12371.612659] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [12372.096060] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12372.107547] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12372.262052] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12382.182698] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12382.371828] Lustre: tst250fs-OST0001: new disk, initializing [12382.374735] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [12382.378802] Lustre: 360085:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12382.459438] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12388.927497] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_default_debug -1 all [12390.970923] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12390.983293] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12391.088189] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12394.641155] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [12397.356817] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [12397.363676] Lustre: Skipped 2 previous similar messages [12418.532962] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12418.537434] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12418.551410] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12422.118313] Lustre: tst250fs-MDT0000-lwp-OST0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12422.119171] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12422.135641] Lustre: Skipped 1 previous similar message [12422.137078] Lustre: Skipped 1 previous similar message [12424.170291] Lustre: server umount tst250fs-MDT0000 complete [12428.396242] LustreError: 358410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1779882030 with bad export cookie 17304489169680689419 [12428.406939] LustreError: MGC192.168.204.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12428.411982] LustreError: 358410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12428.658612] Lustre: server umount tst250fs-MDT0001 complete [12441.664718] Lustre: server umount tst250fs-OST0000 complete [12454.798035] Lustre: server umount tst250fs-OST0001 complete [12476.215922] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [12484.926061] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [12516.988224] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 07:41:57 (1779882117) [12519.050419] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [12537.905414] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing set_hostid [12549.873085] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing load_modules_local [12586.230974] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12341 sec ========= 07:43:06 (1779882186) [12587.903902] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 07:43:08 (1779882188) === [12591.369067] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 07:43:11 (1779882191) === [12631.617154] Lustre: DEBUG MARKER: oleg456-server.virtnet: executing unload_modules_local [12634.492260] Key type lgssc unregistered [12634.889408] LNet: 369134:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12634.908023] LNetError: 369134:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12634.934969] LNet: Removed LNI 192.168.204.156@tcp [12636.004398] Key type .llcrypt unregistered [12636.007430] Key type ._llcrypt unregistered