[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 476141045 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003174] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.011092] LSM: Security Framework initializing [ 0.012059] Yama: becoming mindful. [ 0.013037] SELinux: Initializing. [ 0.014067] *** VALIDATE selinux *** [ 0.022781] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027174] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028150] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029108] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030107] *** VALIDATE tmpfs *** [ 0.032235] *** VALIDATE proc *** [ 0.033252] *** VALIDATE cgroup *** [ 0.035006] *** VALIDATE cgroup2 *** [ 0.036190] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037163] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039029] Spectre V2 : User space: Vulnerable [ 0.040007] Speculative Store Bypass: Vulnerable [ 0.043230] debug: unmapping init [mem 0xffffffff86659000-0xffffffff86660fff] [ 0.045185] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046702] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047020] ... version: 2 [ 0.048012] ... bit width: 48 [ 0.049012] ... generic registers: 4 [ 0.050015] ... value mask: 0000ffffffffffff [ 0.051014] ... max period: 00007fffffffffff [ 0.052012] ... fixed-purpose events: 3 [ 0.053024] ... event mask: 000000070000000f [ 0.054223] rcu: Hierarchical SRCU implementation. [ 0.056467] smp: Bringing up secondary CPUs ... [ 0.057602] x86: Booting SMP configuration: [ 0.058020] .... node #0, CPUs: #1 #2 #3 [ 0.061390] smp: Brought up 1 node, 4 CPUs [ 0.063012] smpboot: Max logical packages: 1 [ 0.064019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.237046] node 0 deferred pages initialised in 170ms [ 0.240294] devtmpfs: initialized [ 0.241224] x86/mm: Memory block size: 128MB [ 0.243857] gcov: version magic: 0x41383552 [ 0.244735] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.249150] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.251474] pinctrl core: initialized pinctrl subsystem [ 0.253213] [ 0.253795] ************************************************************* [ 0.257019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.260018] ** ** [ 0.262019] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.265023] ** ** [ 0.267023] ** This means that this kernel is built to expose internal ** [ 0.269022] ** IOMMU data structures, which may compromise security on ** [ 0.271021] ** your system. ** [ 0.274021] ** ** [ 0.277024] ** If you see this message and you are not debugging the ** [ 0.279019] ** kernel, report this immediately to your vendor! ** [ 0.282018] ** ** [ 0.284020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.287021] ************************************************************* [ 0.290841] NET: Registered protocol family 16 [ 0.292516] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.295100] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.298091] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.301553] cpuidle: using governor menu [ 0.302754] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.305632] PCI: Using configuration type 1 for base access [ 0.308131] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.318126] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.321034] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.325231] cryptd: max_cpu_qlen set to 1000 [ 0.328227] ACPI: Added _OSI(Module Device) [ 0.329044] ACPI: Added _OSI(Processor Device) [ 0.330025] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.332019] ACPI: Added _OSI(Processor Aggregator Device) [ 0.337234] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.345440] ACPI: Interpreter enabled [ 0.347069] ACPI: PM: (supports S0 S3 S4 S5) [ 0.349019] ACPI: Using IOAPIC for interrupt routing [ 0.351273] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.355420] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.367060] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.369047] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.372035] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.375077] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.380424] acpiphp: Slot [2] registered [ 0.382158] acpiphp: Slot [5] registered [ 0.384168] acpiphp: Slot [6] registered [ 0.385125] acpiphp: Slot [7] registered [ 0.387091] acpiphp: Slot [8] registered [ 0.388000] acpiphp: Slot [9] registered [ 0.389102] acpiphp: Slot [10] registered [ 0.391143] acpiphp: Slot [3] registered [ 0.392098] acpiphp: Slot [4] registered [ 0.394123] acpiphp: Slot [11] registered [ 0.395113] acpiphp: Slot [12] registered [ 0.397111] acpiphp: Slot [13] registered [ 0.399142] acpiphp: Slot [14] registered [ 0.400163] acpiphp: Slot [15] registered [ 0.402102] acpiphp: Slot [16] registered [ 0.403088] acpiphp: Slot [17] registered [ 0.405098] acpiphp: Slot [18] registered [ 0.406128] acpiphp: Slot [19] registered [ 0.408114] acpiphp: Slot [20] registered [ 0.410086] acpiphp: Slot [21] registered [ 0.411121] acpiphp: Slot [22] registered [ 0.413099] acpiphp: Slot [23] registered [ 0.414097] acpiphp: Slot [24] registered [ 0.415088] acpiphp: Slot [25] registered [ 0.417105] acpiphp: Slot [26] registered [ 0.418102] acpiphp: Slot [27] registered [ 0.420080] acpiphp: Slot [28] registered [ 0.421098] acpiphp: Slot [29] registered [ 0.423124] acpiphp: Slot [30] registered [ 0.424118] acpiphp: Slot [31] registered [ 0.426095] PCI host bridge to bus 0000:00 [ 0.427015] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.429024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.432024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.434024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.437025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.440029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.442225] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.445064] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.448178] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.457796] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.462523] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.464017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.466014] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.468016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.470627] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.473952] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.477045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.480857] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.488015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.502015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.508014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.515104] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.524032] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.533017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.551018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.563762] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.569013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.578017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.593073] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.608078] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.617017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.624023] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.641017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.649448] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.658016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.667028] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.691017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.705298] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.716024] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.724024] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.745024] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.757065] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.768017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.777022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.797026] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.810024] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.814471] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.816523] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.819460] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.823355] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.828149] iommu: Default domain type: Passthrough [ 0.830493] SCSI subsystem initialized [ 0.832202] ACPI: bus type USB registered [ 0.834182] usbcore: registered new interface driver usbfs [ 0.836128] usbcore: registered new interface driver hub [ 0.839123] usbcore: registered new device driver usb [ 0.841236] pps_core: LinuxPPS API ver. 1 registered [ 0.843016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.846089] PTP clock support registered [ 0.849057] EDAC MC: Ver: 3.0.0 [ 0.851150] PCI: Using ACPI for IRQ routing [ 0.853640] NetLabel: Initializing [ 0.855020] NetLabel: domain hash size = 128 [ 0.857014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.859107] NetLabel: unlabeled traffic allowed by default [ 0.862092] vgaarb: loaded [ 0.864033] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.865016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.870225] clocksource: Switched to clocksource kvm-clock [ 0.978689] VFS: Disk quotas dquot_6.6.0 [ 0.980203] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.982933] *** VALIDATE ramfs *** [ 0.984293] *** VALIDATE hugetlbfs *** [ 0.985777] pnp: PnP ACPI init [ 0.988222] pnp: PnP ACPI: found 6 devices [ 1.006393] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.008842] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.010835] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.012678] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.014964] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.017528] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.021245] NET: Registered protocol family 2 [ 1.024238] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.029934] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.034167] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.040268] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.044124] TCP: Hash tables configured (established 65536 bind 65536) [ 1.047232] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.050622] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.053521] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.056548] NET: Registered protocol family 1 [ 1.059665] RPC: Registered named UNIX socket transport module. [ 1.062279] RPC: Registered udp transport module. [ 1.064342] RPC: Registered tcp transport module. [ 1.066629] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.069852] NET: Registered protocol family 44 [ 1.072087] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.075737] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.078212] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.082439] PCI: CLS 0 bytes, default 64 [ 1.084400] Unpacking initramfs... [ 2.495427] debug: unmapping init [mem 0xffff90ebfcc54000-0xffff90ebfffbffff] [ 2.499795] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.502757] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.506371] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.034362] Initialise system trusted keyrings [ 3.036770] Key type blacklist registered [ 3.038764] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.051361] zbud: loaded [ 3.055605] *** VALIDATE nfs *** [ 3.057546] *** VALIDATE nfs4 *** [ 3.059934] pstore: using deflate compression [ 3.065639] Platform Keyring initialized [ 3.168090] NET: Registered protocol family 38 [ 3.169867] Key type asymmetric registered [ 3.171410] Asymmetric key parser 'x509' registered [ 3.173289] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.176265] io scheduler mq-deadline registered [ 3.177888] io scheduler kyber registered [ 3.179557] io scheduler bfq registered [ 3.181495] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.184484] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.187346] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.190117] ACPI: Power Button [PWRF] [ 3.195754] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.203124] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.217310] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.223187] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.236765] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.265341] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.293892] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.298522] Non-volatile memory driver v1.3 [ 3.300063] Linux agpgart interface v0.103 [ 3.333974] virtio_blk virtio1: [vda] 145896 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.337435] vda: detected capacity change from 0 to 74698752 [ 3.356063] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.359266] vdb: detected capacity change from 0 to 1073741824 [ 3.378638] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.381117] vdc: detected capacity change from 0 to 2621440000 [ 3.397370] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.401108] vdd: detected capacity change from 0 to 2621440000 [ 3.419394] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.423232] vde: detected capacity change from 0 to 4294967296 [ 3.439447] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.442772] vdf: detected capacity change from 0 to 4294967296 [ 3.449530] libphy: Fixed MDIO Bus: probed [ 3.470581] usbcore: registered new interface driver usbserial_generic [ 3.473113] usbserial: USB Serial support registered for generic [ 3.475543] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.479398] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.481625] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.483543] mousedev: PS/2 mouse device common for all mice [ 3.486094] rtc_cmos 00:05: RTC can wake from S4 [ 3.488627] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.491752] rtc_cmos 00:05: registered as rtc0 [ 3.495119] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.498377] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.499408] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.504557] intel_pstate: CPU model not supported [ 3.507993] hid: raw HID events driver (C) Jiri Kosina [ 3.510660] usbcore: registered new interface driver usbhid [ 3.513284] usbhid: USB HID core driver [ 3.515167] drop_monitor: Initializing network drop monitor service [ 3.517767] Initializing XFRM netlink socket [ 3.519715] NET: Registered protocol family 10 [ 3.522902] Segment Routing with IPv6 [ 3.524270] NET: Registered protocol family 17 [ 3.525897] mpls_gso: MPLS GSO support [ 3.531212] RAS: Correctable Errors collector initialized. [ 3.533719] AVX version of gcm_enc/dec engaged. [ 3.535419] AES CTR mode by8 optimization enabled [ 3.607747] sched_clock: Marking stable (3607725164, 0)->(4532309073, -924583909) [ 3.611114] registered taskstats version 1 [ 3.612907] Loading compiled-in X.509 certificates [ 3.614954] zswap: loaded using pool lzo/zbud [ 3.636455] Key type big_key registered [ 3.648274] Key type encrypted registered [ 3.649940] ima: No TPM chip found, activating TPM-bypass! [ 3.651959] ima: Allocated hash algorithm: sha1 [ 3.653503] ima: No architecture policies found [ 3.655311] evm: Initialising EVM extended attributes: [ 3.657216] evm: security.selinux [ 3.658362] evm: security.ima [ 3.659375] evm: security.capability [ 3.660635] evm: HMAC attrs: 0x1 [ 3.662820] rtc_cmos 00:05: setting system clock to 2026-09-03 13:29:14 UTC (1788442154) [ 3.668981] debug: unmapping init [mem 0xffffffff87603000-0xffffffff877fffff] [ 3.671841] debug: unmapping init [mem 0xffffffff86382000-0xffffffff86658fff] [ 3.681145] Write protecting the kernel read-only data: 28672k [ 3.684043] debug: unmapping init [mem 0xffffffff84a03000-0xffffffff84bfffff] [ 3.686557] debug: unmapping init [mem 0xffffffff85314000-0xffffffff853fffff] [ 3.718729] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.727147] systemd[1]: Detected virtualization kvm. [ 3.729105] systemd[1]: Detected architecture x86-64. [ 3.731838] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.759159] systemd[1]: No hostname configured. [ 3.761186] systemd[1]: Set hostname to . [ 3.763061] random: systemd: uninitialized urandom read (16 bytes read) [ 3.765795] systemd[1]: Initializing machine ID from random generator. [ 3.845645] random: ln: uninitialized urandom read (6 bytes read) [ 3.929474] random: systemd: uninitialized urandom read (16 bytes read) [ 3.931594] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.936493] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.946294] systemd[1]: Started Memstrack Anylazing Service. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. [ OK ] Reached target Local Encrypted Volumes. Starting Setup Virtual Console... [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Reached target Timers. Starting Apply Kernel Variables... [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Slices. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.578706] device-mapper: uevent: version 1.0.3 [ 4.581337] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [[ 4.965395] random: fast init done  OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.372782] virtio_net virtio0 ens2: renamed from eth0 [ 5.447298] scsi host0: ata_piix [ 5.478171] scsi host1: ata_piix [ 5.482272] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.485075] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.275774] dracut-initqueue[579]: RTNETLINK answers: File exists [ 10.165251] random: crng init done [ 10.167216] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.708739] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.933475] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.221716] SELinux: Disabled at runtime. [ 12.284689] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.293818] systemd[1]: Detected virtualization kvm. [ 12.295996] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.829795] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.833727] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.838652] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.844379] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.848104] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.865392] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.870208] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Created slice system-getty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Local Encrypted Volumes. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice User and Session Slice. [ 13.029104] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.350176] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.721106] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.738716] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.846809] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.884101] EDAC sbridge: Ver: 1.1.2 [ 15.270264] Key type dns_resolver registered [ 15.584025] NFS: Registering the id_resolver key type [ 15.586044] Key type id_resolver registered [ 15.587369] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started dnf makecache --timer. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. [ OK ] Started daily update of the root trust anchor for DNSSEC. Starting Network Manager... Starting Login Service... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg651-server login: [ 45.456815] libcfs: loading out-of-tree module taints kernel. [ 45.510900] Key type ._llcrypt registered [ 45.513656] Key type .llcrypt registered [ 45.599642] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 62.655433] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 64.101834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 64.113912] alg: No test for adler32 (adler32-zlib) [ 65.557042] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 66.280473] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 68.095249] Key type lgssc registered [ 69.690357] Lustre: Echo OBD driver; http://www.lustre.org/ [ 86.365782] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 122.202511] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 134.143709] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 134.199749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 135.506674] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 135.540377] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 135.648376] Lustre: lustre-MDT0000: new disk, initializing [ 135.761680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 135.793453] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 140.241024] hrtimer: interrupt took 2540684 ns [ 140.722315] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 153.270436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 153.371464] Lustre: 6511:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 153.420952] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 153.426519] Lustre: Skipped 1 previous similar message [ 153.508845] Lustre: lustre-MDT0001: new disk, initializing [ 153.611876] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 153.637423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 153.644809] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 157.773387] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 162.502040] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 172.345808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 172.689322] Lustre: lustre-OST0000: new disk, initializing [ 172.693284] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 172.700956] Lustre: 8450:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 172.760967] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 175.920673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 175.937295] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 176.068221] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 179.091658] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 193.493311] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 193.611920] Lustre: lustre-OST0001: new disk, initializing [ 193.615287] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 193.620898] Lustre: 9523:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 193.669634] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 199.245306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 199.259152] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 199.343523] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 200.636812] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 212.858718] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 219.456039] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 226.094899] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing check_logdir /tmp/testlogs/ [ 230.957545] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing yml_node [ 234.762356] Lustre: DEBUG MARKER: Client: 2.17.56.51 [ 236.973493] Lustre: DEBUG MARKER: MDS: 2.17.56.51 [ 239.538759] Lustre: DEBUG MARKER: OSS: 2.17.56.51 [ 241.054658] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 09:33:10 EDT 2026 [ 257.476388] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 259.125298] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 276.450982] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 276.454961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 276.460088] Lustre: Skipped 2 previous similar messages [ 276.461725] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 276.467482] Lustre: Skipped 3 previous similar messages [ 280.732657] Lustre: server umount lustre-MDT0000 complete [ 284.341327] LustreError: 6505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442435 with bad export cookie 5237223999640422692 [ 284.345586] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 284.347297] LustreError: 6505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 286.689427] LustreError: 6521:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 286.697134] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 286.721035] LustreError: 6521:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 286.750478] Lustre: Skipped 1 previous similar message [ 286.751054] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 289.698328] LustreError: 6519:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 289.699063] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 289.718184] Lustre: Skipped 2 previous similar messages [ 290.818936] Lustre: server umount lustre-MDT0001 complete [ 294.870535] Lustre: server umount lustre-OST0000 complete [ 298.967307] Lustre: server umount lustre-OST0001 complete [ 305.835280] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 315.082586] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 350.301073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 350.685358] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 350.732505] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 350.895743] Lustre: lustre-MDT0000: new disk, initializing [ 350.998929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 351.014361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 355.720682] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 365.803934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 365.899490] Lustre: 15071:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 365.948790] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 365.955227] Lustre: Skipped 1 previous similar message [ 366.038709] Lustre: lustre-MDT0001: new disk, initializing [ 366.161510] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 366.231598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 366.244793] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 370.285663] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 379.921744] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 386.641764] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 394.074668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 394.333518] Lustre: lustre-OST0000: new disk, initializing [ 394.336874] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 394.342849] Lustre: 17044:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 394.427556] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 395.770265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 395.777912] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 395.902943] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 400.679171] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 410.718641] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 415.016402] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 415.237566] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 419.754526] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 419.996358] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 426.483910] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 426.491084] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 426.493269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 426.530403] Lustre: Skipped 2 previous similar messages [ 428.445685] Lustre: server umount lustre-OST0000 complete [ 432.610079] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 432.619300] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 432.630260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 432.635253] Lustre: Skipped 1 previous similar message [ 436.708612] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 438.483704] Lustre: server umount lustre-MDT0000 complete [ 441.833183] LustreError: 15083:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 441.864741] LustreError: 15083:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 442.227832] LustreError: 15065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442593 with bad export cookie 5237223999640424225 [ 442.229945] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 442.245971] LustreError: 15065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 442.603802] Lustre: server umount lustre-MDT0001 complete [ 448.270655] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 09:36:37 (1788442597) [ 466.242657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 466.882463] LustreError: 19918:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 466.978480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 471.421320] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 472.034045] LustreError: 19919:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 480.501304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 486.518986] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 494.818307] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 500.884505] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 503.686296] Lustre: Failing over lustre-MDT0000 [ 504.036995] Lustre: server umount lustre-MDT0000 complete [ 505.762218] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 505.787220] LustreError: 19919:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 505.836228] LustreError: 19919:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 522.079091] Lustre: 3647:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788442656/real 1788442656] req@ffff90eb4437df80 x1875317586290304/t0(0) o400->MGC192.168.206.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788442672 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 522.127802] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 522.144217] LustreError: 19919:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 522.157031] LustreError: 19919:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 524.067884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 531.369899] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x48ae5b7acbb7d4a7 [ 531.393081] Lustre: MGC192.168.206.151@tcp: Connection restored to 0@lo (at 0@lo) [ 531.643556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 531.651973] Lustre: Skipped 1 previous similar message [ 536.320522] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 537.065461] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 537.068158] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 537.134151] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 546.576351] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 552.417300] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 552.425426] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 552.455698] Lustre: Skipped 1 previous similar message [ 552.470137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 552.481067] Lustre: Skipped 2 previous similar messages [ 559.614956] Lustre: server umount lustre-MDT0000 complete [ 560.610463] LustreError: 19918:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 560.640054] LustreError: 19918:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 563.688773] LustreError: 19898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442714 with bad export cookie 5237223999640425639 [ 563.703837] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 563.707482] LustreError: 19898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 564.126127] Lustre: server umount lustre-MDT0001 complete [ 572.973439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 579.067810] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 589.022863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 594.927452] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 603.472504] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 610.152641] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 612.273852] Lustre: Failing over lustre-MDT0000 [ 612.462710] Lustre: server umount lustre-MDT0000 complete [ 614.378310] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 614.393704] Lustre: Skipped 1 previous similar message [ 624.611423] LustreError: 23261:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 624.637594] LustreError: 23261:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 630.751635] Lustre: 3649:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788442765/real 1788442765] req@ffff90ec44acc380 x1875317586320000/t0(0) o400->MGC192.168.206.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788442781 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 630.787235] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 631.643469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 642.443378] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 646.767985] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 647.653826] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 647.658582] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 647.669450] Lustre: Skipped 1 previous similar message [ 647.710491] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 656.511487] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 663.008494] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 663.023681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 663.025726] Lustre: Skipped 2 previous similar messages [ 666.517500] Lustre: server umount lustre-MDT0000 complete [ 670.185992] LustreError: 3646:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff90eb44365c00 x1875317586334208/t0(0) o38->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 670.186848] LustreError: 23240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442821 with bad export cookie 5237223999640426584 [ 670.200969] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 670.248292] LustreError: 23240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 670.553877] Lustre: server umount lustre-MDT0001 complete [ 680.443411] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 09:40:29 (1788442829) [ 697.981671] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 709.651442] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 753.153459] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 764.830597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 765.165041] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 765.187932] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 765.285508] Lustre: lustre-MDT0000: new disk, initializing [ 765.390510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 765.398940] Lustre: Skipped 3 previous similar messages [ 765.416078] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 770.254849] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 781.084891] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 781.226873] Lustre: 29636:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 781.461421] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 786.069887] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 796.332031] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 802.767968] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 809.699279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 809.943859] Lustre: 31606:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 811.555352] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 811.588744] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 811.641558] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 811.660898] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 816.444971] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 827.239462] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 832.966752] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 857.569735] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 857.585254] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 857.594712] Lustre: Skipped 3 previous similar messages [ 857.603222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 857.609783] Lustre: Skipped 2 previous similar messages [ 860.443183] Lustre: server umount lustre-OST0000 complete [ 870.294243] Lustre: server umount lustre-MDT0000 complete [ 873.443609] LustreError: 30526:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 873.465527] LustreError: 30526:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 874.214860] LustreError: 29626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443025 with bad export cookie 5237223999640426990 [ 874.230185] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 874.231452] LustreError: 29626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 874.726316] Lustre: server umount lustre-MDT0001 complete [ 886.138825] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 888.958499] Key type lgssc unregistered [ 889.462041] LNet: 33589:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 889.491957] LNetError: 33589:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 889.530761] LNet: Removed LNI 192.168.206.151@tcp [ 890.795150] Key type .llcrypt unregistered [ 890.798576] Key type ._llcrypt unregistered [ 915.937959] Key type ._llcrypt registered [ 915.942406] Key type .llcrypt registered [ 916.081510] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 917.211242] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 917.253329] alg: No test for adler32 (adler32-zlib) [ 918.390635] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 918.648648] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 920.343178] Key type lgssc registered [ 921.335532] Lustre: Echo OBD driver; http://www.lustre.org/ [ 930.195523] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 930.234515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 931.792270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 936.323339] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 942.060120] LustreError: 35003:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 942.080848] LustreError: 35003:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 946.375036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 946.902326] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 952.417057] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 961.382310] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 968.404489] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 975.370476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 975.771467] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 982.923947] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 991.823316] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 992.194794] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1007.007845] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 09:45:55 (1788443155) [ 1009.082563] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1011.189704] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1018.153983] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 09:46:07 (1788443167) [ 1027.573545] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 09:46:16 (1788443176) [ 1028.510966] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1036.777610] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1036.782556] Lustre: Skipped 2 previous similar messages [ 1038.374913] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 09:46:27 (1788443187) [ 1042.385589] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1042.389728] Lustre: Skipped 3 previous similar messages [ 1059.186317] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1059.190907] Lustre: Skipped 6 previous similar messages [ 1062.086937] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 09:46:50 (1788443210) [ 1064.612450] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1068.096685] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1076.287993] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 09:47:05 (1788443225) [ 1078.325504] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1078.331789] Lustre: Skipped 3 previous similar messages [ 1090.968244] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 09:47:19 (1788443239) [ 1104.864086] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1104.865252] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1104.877883] LustreError: Skipped 1 previous similar message [ 1104.882187] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1104.897765] Lustre: Skipped 1 previous similar message [ 1104.914048] Lustre: Skipped 1 previous similar message [ 1109.721165] Lustre: server umount lustre-OST0000 complete [ 1115.105221] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1115.106832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1115.117145] Lustre: Skipped 1 previous similar message [ 1115.124767] Lustre: Skipped 1 previous similar message [ 1119.750751] Lustre: server umount lustre-MDT0000 complete [ 1120.235560] LustreError: 35003:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1120.252554] LustreError: 35003:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1123.541193] LustreError: 34983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443274 with bad export cookie 15432714643296542887 [ 1123.548408] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1123.561788] LustreError: 34983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1124.109922] Lustre: server umount lustre-MDT0001 complete [ 1134.444757] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 1137.028413] Key type lgssc unregistered [ 1137.320918] LNet: 42853:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1137.335894] LNetError: 42853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1137.360825] LNet: Removed LNI 192.168.206.151@tcp [ 1138.207896] Key type .llcrypt unregistered [ 1138.209748] Key type ._llcrypt unregistered [ 1156.020519] Key type ._llcrypt registered [ 1156.023402] Key type .llcrypt registered [ 1156.134066] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 1170.689399] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 1171.704658] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1171.803984] alg: No test for adler32 (adler32-zlib) [ 1172.905838] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1173.139049] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 1174.816822] Key type lgssc registered [ 1175.832876] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1212.952762] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1212.970492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1214.230480] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1214.281524] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1214.383637] Lustre: lustre-MDT0000: new disk, initializing [ 1214.448935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1214.484357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1219.097726] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1229.575423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1229.668500] Lustre: 46010:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1229.774145] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1229.780406] Lustre: Skipped 1 previous similar message [ 1229.783218] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1229.862360] Lustre: lustre-MDT0001: new disk, initializing [ 1229.972803] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1230.017143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1230.030370] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1233.994630] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1243.677788] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1249.899787] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1257.339937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1257.739847] Lustre: lustre-OST0000: new disk, initializing [ 1257.743266] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1257.749766] Lustre: 47976:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1257.831362] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1263.748633] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1265.712760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1265.722017] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1265.804491] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1272.261432] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1277.409538] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1277.711786] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1282.289388] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1282.460903] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1286.112877] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1286.121732] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1286.138818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1286.624499] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1286.634222] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1290.645642] Lustre: server umount lustre-OST0000 complete [ 1296.352427] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1296.362283] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1296.389667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1296.401569] Lustre: Skipped 1 previous similar message [ 1301.992755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1302.004883] Lustre: Skipped 2 previous similar messages [ 1307.104327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1307.114560] Lustre: Skipped 1 previous similar message [ 1309.151981] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1309.372204] Lustre: server umount lustre-MDT0000 complete [ 1313.497245] LustreError: 46805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443464 with bad export cookie 11698394580196849358 [ 1313.504527] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1313.507726] LustreError: 46805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1313.895738] Lustre: server umount lustre-MDT0001 complete [ 1323.878527] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 09:51:12 (1788443472) [ 1331.330311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1331.827386] LustreError: 50439:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1331.844897] LustreError: 50439:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1331.906961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1335.927454] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1337.312511] LustreError: 50438:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1341.412906] LustreError: 50439:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1343.606419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1343.905515] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1348.129560] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1354.958526] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1360.274583] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1366.719058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1367.047224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1373.055244] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1380.717301] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1385.522986] LustreError: 52967:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1386.540752] Lustre: Modifying parameter general.timeout=1 in log params [ 1388.683492] Lustre: Modifying parameter general.timeout=3 in log params [ 1388.690344] Lustre: Skipped 1 previous similar message [ 1390.932859] Lustre: Modifying parameter general.timeout=5 in log params [ 1390.945385] Lustre: Skipped 1 previous similar message [ 1395.250992] Lustre: Modifying parameter general.timeout=9 in log params [ 1395.257960] Lustre: Skipped 3 previous similar messages [ 1403.485129] Lustre: Modifying parameter general.timeout=18 in log params [ 1403.488702] Lustre: Skipped 8 previous similar messages [ 1420.020157] Lustre: Modifying parameter general.timeout=35 in log params [ 1420.027649] Lustre: Skipped 16 previous similar messages [ 1452.506392] Lustre: Modifying parameter general.timeout=70 in log params [ 1452.508890] Lustre: Skipped 34 previous similar messages [ 1519.111969] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 09:54:28 (1788443668) [ 1520.167946] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_policies=tbf in log params [ 1520.175984] Lustre: Skipped 60 previous similar messages [ 1522.284131] Lustre: 59911:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1525.132297] Lustre: 60019:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1527.461731] Lustre: 60119:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1534.472918] Lustre: 60430:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1534.482275] Lustre: 60430:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1565.403166] Lustre: 61785:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1565.434834] LustreError: 61785:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1569.240663] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 09:55:18 (1788443718) [ 1584.609081] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1584.625086] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1584.650295] Lustre: Skipped 1 previous similar message [ 1584.658161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1587.665307] Lustre: server umount lustre-MDT0000 complete [ 1590.996873] LustreError: 50420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443741 with bad export cookie 11698394580196850240 [ 1591.000434] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1591.007911] LustreError: 50420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1591.270441] LustreError: 50434:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1591.276374] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1591.310417] Lustre: Skipped 2 previous similar messages [ 1591.485893] Lustre: server umount lustre-MDT0001 complete [ 1597.172928] Lustre: server umount lustre-OST0000 complete [ 1622.757992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1623.085462] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1623.486524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1628.228029] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1636.876246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1636.950349] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1636.981315] Lustre: 64315:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1641.608492] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1649.251117] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1654.521239] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1661.632476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1661.808230] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1662.143204] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1662.149967] Lustre: Skipped 1 previous similar message [ 1668.042428] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1675.467744] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1677.367579] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1677.381921] Lustre: Skipped 22 previous similar messages [ 1677.431205] Lustre: 66880:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1688.033067] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1688.045816] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1688.065726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1688.070287] Lustre: Skipped 3 previous similar messages [ 1690.021946] Lustre: server umount lustre-OST0000 complete [ 1699.781617] Lustre: server umount lustre-MDT0000 complete [ 1703.194173] LustreError: 64307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443854 with bad export cookie 11698394580196853355 [ 1703.199374] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1703.207692] LustreError: 64307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1703.702436] Lustre: server umount lustre-MDT0001 complete [ 1713.987968] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 1716.774709] Key type lgssc unregistered [ 1717.129750] LNet: 68094:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1717.136947] LNetError: 68094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1717.152231] LNet: Removed LNI 192.168.206.151@tcp [ 1718.071105] Key type .llcrypt unregistered [ 1718.075743] Key type ._llcrypt unregistered [ 1742.602962] Key type ._llcrypt registered [ 1742.605749] Key type .llcrypt registered [ 1742.716321] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 1743.558220] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1743.567889] alg: No test for adler32 (adler32-zlib) [ 1744.631178] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1744.847990] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 1746.527184] Key type lgssc registered [ 1747.830447] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1755.190531] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1755.221093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1756.685405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1760.681718] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1761.761459] LustreError: 69509:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1761.784145] LustreError: 69509:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1766.882298] LustreError: 69510:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1769.048537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1769.550918] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1773.831830] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1780.630784] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1786.153696] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1793.155176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1794.094035] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1800.435786] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1808.231592] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1817.344972] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1830.881671] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1830.911797] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1830.956771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1831.907198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1831.924872] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1834.412617] Lustre: server umount lustre-MDT0000 complete [ 1837.033203] LustreError: 69509:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1837.068992] LustreError: 69509:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1840.628237] LustreError: 70249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443991 with bad export cookie 10695217853257413389 [ 1840.641091] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1840.653444] LustreError: 70249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1841.343978] Lustre: server umount lustre-MDT0001 complete [ 1858.766493] Lustre: server umount lustre-OST0000 complete [ 1885.908628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1886.172380] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1886.202306] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1886.470829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1891.174913] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1900.551316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1900.646095] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1900.702184] Lustre: 74381:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1901.284670] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1907.334904] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1915.536156] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1921.610592] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1928.586381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1928.812798] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1929.073209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1935.139461] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1941.783151] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1943.531779] Lustre: 76969:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1945.461545] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1955.298852] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1955.315516] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1955.349555] Lustre: Skipped 1 previous similar message [ 1955.366366] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1955.389108] Lustre: Skipped 2 previous similar messages [ 1957.681790] Lustre: server umount lustre-OST0000 complete [ 1966.055178] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1966.074501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1966.074649] Lustre: Skipped 2 previous similar messages [ 1966.095431] Lustre: Skipped 2 previous similar messages [ 1969.886631] Lustre: server umount lustre-MDT0000 complete [ 1971.167995] LustreError: 74388:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1971.205994] LustreError: 74388:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1973.613637] LustreError: 76275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444124 with bad export cookie 10695217853257414243 [ 1973.619938] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1973.624911] LustreError: 76275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1973.907283] Lustre: server umount lustre-MDT0001 complete [ 1983.912403] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 1986.625876] Key type lgssc unregistered [ 1986.879308] LNet: 78185:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1986.892862] LNetError: 78185:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1987.944624] LNet: Removed LNI 192.168.206.151@tcp [ 1988.772674] Key type .llcrypt unregistered [ 1988.774471] Key type ._llcrypt unregistered [ 2010.770967] Key type ._llcrypt registered [ 2010.775132] Key type .llcrypt registered [ 2010.913434] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 2011.768627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2011.852123] alg: No test for adler32 (adler32-zlib) [ 2012.916677] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2013.171591] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 2014.824378] Key type lgssc registered [ 2015.869846] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2024.445752] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2024.473367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2026.239218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2030.284275] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2035.699302] LustreError: 79598:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2035.722773] LustreError: 79598:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2038.201471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2038.843393] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2043.837621] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2051.324467] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2056.875574] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2063.994330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2064.717982] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2072.451866] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2080.995639] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2087.911779] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2089.108609] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2091.206421] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2091.216631] Lustre: Skipped 1 previous similar message [ 2094.429761] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2094.440953] Lustre: Skipped 2 previous similar messages [ 2099.110899] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2099.120200] Lustre: Skipped 4 previous similar messages [ 2107.319451] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2107.326635] Lustre: Skipped 8 previous similar messages [ 2124.074613] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2124.077419] Lustre: Skipped 15 previous similar messages [ 2156.830161] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2156.837406] Lustre: Skipped 30 previous similar messages [ 2200.154584] Lustre: 87711:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2202.621783] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 10:05:51 (1788444351) [ 2204.594338] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2206.704463] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 10:05:55 (1788444355) [ 2214.372204] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2214.384654] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2214.395947] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2216.417442] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2216.426984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2216.433105] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2216.464680] Lustre: Skipped 1 previous similar message [ 2221.543148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2226.144143] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2226.338177] Lustre: server umount lustre-OST0000 complete [ 2231.783426] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2231.792793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2231.814673] Lustre: Skipped 1 previous similar message [ 2231.831062] Lustre: Skipped 2 previous similar messages [ 2236.859581] Lustre: server umount lustre-MDT0000 complete [ 2236.900298] LustreError: 79599:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2236.917789] LustreError: 79599:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2240.895478] LustreError: 86556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444391 with bad export cookie 5761708784949538953 [ 2240.898886] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2240.908867] LustreError: 86556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2241.192901] Lustre: server umount lustre-MDT0001 complete [ 2252.419581] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 2255.612312] Key type lgssc unregistered [ 2255.958533] LNet: 88964:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2255.973342] LNetError: 88964:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2255.992145] LNet: Removed LNI 192.168.206.151@tcp [ 2257.003737] Key type .llcrypt unregistered [ 2257.006831] Key type ._llcrypt unregistered [ 2262.516309] Key type ._llcrypt registered [ 2262.522303] Key type .llcrypt registered [ 2262.709798] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2269.918710] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules [ 2271.622602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2271.654637] alg: No test for adler32 (adler32-zlib) [ 2272.851368] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2273.164236] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 2275.007870] Key type lgssc registered [ 2276.569736] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2278.216118] LustreError: 90222:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2293.928732] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 2299.654660] LustreError: 90222:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2303.766053] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2303.815897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2305.565187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2311.020284] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2315.747204] LustreError: 91239:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2315.778386] LustreError: 91239:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2320.722087] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 10:07:49 (1788444469) [ 2320.866804] LustreError: 91240:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2325.453077] LustreError: 91271:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2325.462908] LustreError: 91271:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2325.470465] LustreError: 91271:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 2325.817534] Lustre: server umount lustre-MDT0000 complete [ 2339.789553] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 2342.660242] Key type lgssc unregistered [ 2342.991217] LNet: 92769:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2343.005749] LNetError: 92769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2343.027412] LNet: Removed LNI 192.168.206.151@tcp [ 2344.046429] Key type .llcrypt unregistered [ 2344.054807] Key type ._llcrypt unregistered [ 2363.275067] Key type ._llcrypt registered [ 2363.279693] Key type .llcrypt registered [ 2363.468371] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 2364.462438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2364.598977] alg: No test for adler32 (adler32-zlib) [ 2365.678855] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2365.905902] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 2367.575248] Key type lgssc registered [ 2368.666474] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2376.485532] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2376.516951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2377.913199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2382.095550] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2387.426803] LustreError: 93987:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2387.452303] LustreError: 93987:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2389.771049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2390.456400] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2394.593288] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2401.566448] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2407.035345] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2413.068813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2413.841662] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2419.599213] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2428.840991] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2486.592215] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 10:10:35 (1788444635) [ 2488.313836] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2490.063613] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 10:10:39 (1788444639) [ 2494.946637] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2494.949043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2494.963854] Lustre: Skipped 1 previous similar message [ 2500.072766] LustreError: 96859:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2500.101511] LustreError: 96859:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2500.162200] Lustre: server umount lustre-MDT0000 complete [ 2503.759795] LustreError: 94726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444654 with bad export cookie 11195723731661255876 [ 2503.761715] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2503.772853] LustreError: 94726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2504.125894] Lustre: server umount lustre-MDT0001 complete [ 2518.713624] Lustre: server umount lustre-OST0000 complete [ 2530.166373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2530.521453] LustreError: 98284:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2530.674240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2535.034392] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2535.906884] LustreError: 98285:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2541.033366] LustreError: 98284:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2543.775245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2544.611317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2548.920931] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2556.854036] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2562.492174] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2576.941286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2577.336123] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2577.343832] LustreError: 98274:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2577.376772] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2577.391551] LustreError: 100366:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2577.400858] LustreError: 100366:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2577.407801] LustreError: 100366:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2577.553847] Lustre: server umount lustre-OST0000 complete [ 2577.555911] LustreError: 100366:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2586.790458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2587.013714] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2587.021572] LustreError: 98272:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2587.057855] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2587.068311] LustreError: 100833:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2587.083568] LustreError: 100833:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2587.090312] LustreError: 100833:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2587.180040] Lustre: server umount lustre-OST0000 complete [ 2587.182802] LustreError: 100833:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2596.463789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2596.690712] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2596.696169] Lustre: Found index 0 for lustre-OST0000, updating log [ 2596.701486] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2596.779434] Lustre: lustre-OST0000: new disk, initializing [ 2596.787181] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2596.793282] Lustre: 101368:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2596.978274] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2603.186084] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2606.656781] Lustre: Failing over lustre-OST0000 [ 2606.811641] Lustre: server umount lustre-OST0000 complete [ 2610.658265] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2610.665430] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2610.681267] Lustre: Skipped 1 previous similar message [ 2610.686134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2610.690815] Lustre: Skipped 2 previous similar messages [ 2611.680781] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2611.688053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2611.725412] Lustre: Skipped 1 previous similar message [ 2613.797411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2613.807745] Lustre: Skipped 1 previous similar message [ 2618.848238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2620.994611] Lustre: server umount lustre-MDT0000 complete [ 2623.970556] LustreError: 98303:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2623.986510] LustreError: 98303:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2624.273475] LustreError: 101367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444775 with bad export cookie 11195723731661258620 [ 2624.287193] LustreError: 101367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2624.288289] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2624.619102] Lustre: server umount lustre-MDT0001 complete [ 2632.961351] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 10:13:02 (1788444782) [ 2652.020983] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 2663.601434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2663.914030] LustreError: 103712:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2664.040171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2669.540859] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2678.133165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2683.791968] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2687.149273] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2693.783122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2694.479945] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2700.387319] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2703.662712] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2703.667458] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2703.676674] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2708.685744] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2708.774645] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:04 [ 2708.786374] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2708.795870] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2708.806111] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2708.806361] Lustre: lustre-OST0000: Recovery over after 0:05, of 1 clients 1 recovered and 0 were evicted. [ 2708.815560] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2708.922424] Lustre: lustre-OST0001: new disk, initializing [ 2708.925693] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2708.931229] Lustre: 106032:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2709.146875] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2709.162504] Lustre: Skipped 2 previous similar messages [ 2714.929139] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2719.327949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2719.339871] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2719.448674] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2722.721636] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2729.625958] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2735.075575] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2735.100585] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2735.110983] Lustre: Skipped 1 previous similar message [ 2735.112025] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2735.126776] Lustre: Skipped 2 previous similar messages [ 2740.724949] Lustre: server umount lustre-MDT0001 complete [ 2741.201764] LustreError: 103707:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2741.231210] LustreError: 103707:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2751.046046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2751.194583] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2751.206952] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2751.321725] Lustre: 103699:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2753.014538] LustreError: 107569:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2753.026848] LustreError: 107569:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2753.071878] LustreError: 103704:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2753.177594] LustreError: 107579:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.206.151@tcp: cfg command failed: rc = -17 [ 2753.187062] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2753.187062] [ 2757.635125] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 2758.609636] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2768.732284] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 10:15:17 (1788444917) [ 2787.449683] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 2805.997582] Lustre: 109647:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2825.541178] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2828.438074] Lustre: 110781:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2925.535620] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2925.551336] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2925.575840] Lustre: Skipped 1 previous similar message [ 2925.591591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2925.601300] Lustre: Skipped 5 previous similar messages [ 2930.064216] Lustre: server umount lustre-MDT0000 complete [ 2931.683067] LustreError: 103707:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2931.697905] LustreError: 103707:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 2933.295632] LustreError: 104855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445084 with bad export cookie 11195723731661259621 [ 2933.300633] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2933.312374] LustreError: 104855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 2933.594077] Lustre: server umount lustre-MDT0001 complete [ 2948.122742] Lustre: server umount lustre-OST0000 complete [ 2963.097068] Lustre: server umount lustre-OST0001 complete [ 2969.074445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2978.675702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2987.918683] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2999.835481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3018.939986] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 3031.184961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3031.205660] Lustre: lustre-MDT0000: reset Object Index mappings [ 3032.000542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3032.022597] Lustre: Skipped 1 previous similar message [ 3037.535390] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3045.445831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3045.466991] Lustre: lustre-MDT0001: reset Object Index mappings [ 3050.517899] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3052.976885] Lustre: 115885:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3059.322448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3066.428328] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3070.203092] LustreError: 116238:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3070.231277] LustreError: 116238:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3070.231451] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 3076.637477] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:381 to 0x280000401:418) [ 3077.564782] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3083.803840] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:157 to 0x2c0000401:193) [ 3083.816780] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:199 to 0x2c0000400:225) [ 3088.488984] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3099.322341] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3103.814458] Lustre: 117762:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3155.538263] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 10:21:44 (1788445304) [ 3184.097399] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3184.102385] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3184.110265] Lustre: Skipped 3 previous similar messages [ 3184.113801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3184.117447] Lustre: Skipped 4 previous similar messages [ 3187.444529] Lustre: server umount lustre-MDT0000 complete [ 3189.805520] LustreError: 114716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445340 with bad export cookie 11195723731661306780 [ 3189.805808] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3189.814491] LustreError: 114716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3196.263290] Lustre: server umount lustre-MDT0001 complete [ 3199.535504] LustreError: 118517:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0000_UUID: namespace resource [0x280000401:0x13d:0x0].0x0 (ffff90ec76c8f800) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 3205.694618] Lustre: server umount lustre-OST0000 complete [ 3215.867950] Lustre: server umount lustre-OST0001 complete [ 3223.558615] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 3232.672324] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 3273.097879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3273.523547] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3273.587778] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3273.764664] Lustre: lustre-MDT0000: new disk, initializing [ 3273.920619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3273.923063] Lustre: Skipped 3 previous similar messages [ 3273.962382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3281.272563] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3290.562534] Lustre: server umount lustre-MDT0000 complete [ 3301.310025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3301.700593] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3301.737697] Lustre: 122038:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3301.770503] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3307.346459] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3312.956041] Lustre: server umount lustre-MDT0000 complete [ 3322.819067] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 10:24:31 (1788445471) [ 3324.755501] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3327.152283] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 10:24:35 (1788445475) [ 3346.689954] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 3356.680174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3361.685938] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3371.432262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3371.569879] Lustre: 124088:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3371.635928] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3371.641026] Lustre: Skipped 1 previous similar message [ 3371.781278] Lustre: lustre-MDT0001: new disk, initializing [ 3371.987278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3372.001268] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3377.163718] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3383.194630] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3391.021094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3391.331776] Lustre: lustre-OST0000: new disk, initializing [ 3391.337133] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3391.345481] Lustre: 125667:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3392.988226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3393.016510] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3393.136665] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3398.710347] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3410.867716] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3411.099623] Lustre: lustre-OST0001: new disk, initializing [ 3411.103962] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3411.110153] Lustre: 126538:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3412.696070] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3412.713637] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3412.838082] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3418.568831] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3428.933263] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3432.877647] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3845.497330] Lustre: 132958:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3845.509280] Lustre: 132958:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3845.521714] Lustre: 132958:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3845.533709] Lustre: 132958:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3845.537887] Lustre: 132958:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3845.544757] Lustre: 132958:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4110.483996] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 10:37:38 (1788446258) [ 4129.770671] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4129.772715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4129.784085] Lustre: Skipped 7 previous similar messages [ 4129.795056] Lustre: Skipped 10 previous similar messages [ 4131.999064] Lustre: server umount lustre-MDT0000 complete [ 4134.885365] LustreError: 124095:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4134.892274] LustreError: 124095:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 4135.711126] LustreError: 124082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446286 with bad export cookie 11195723731661375583 [ 4135.716246] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4135.721774] LustreError: 124082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4140.000599] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4140.006262] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4140.022344] Lustre: Skipped 2 previous similar messages [ 4150.239133] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4150.598156] Lustre: server umount lustre-MDT0001 complete [ 4160.957754] Lustre: server umount lustre-OST0000 complete [ 4170.981575] Lustre: server umount lustre-OST0001 complete [ 4178.057347] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 4187.826876] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 4220.706918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4220.926893] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4220.958377] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4221.046339] Lustre: lustre-MDT0000: new disk, initializing [ 4221.140109] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4221.148856] Lustre: Skipped 5 previous similar messages [ 4221.166596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4225.379856] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4236.456262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4236.601243] Lustre: 136006:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4236.698500] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4236.702299] Lustre: Skipped 1 previous similar message [ 4236.814557] Lustre: lustre-MDT0001: new disk, initializing [ 4236.942771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4236.955659] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4242.041580] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4252.213911] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4260.452739] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4267.769581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4268.386760] Lustre: lustre-OST0000: new disk, initializing [ 4268.400175] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4268.413516] Lustre: 137977:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4268.581716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4268.589281] Lustre: Skipped 1 previous similar message [ 4270.062392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4270.078276] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4270.282458] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4275.552793] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4285.172861] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4294.586197] Lustre: lustre-MDD0000: changelog on [ 4296.854647] Lustre: lustre-MDD0001: changelog on [ 6220.228620] Lustre: lustre-MDD0001: changelog off [ 6222.477176] Lustre: 136015:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6223.549329] Lustre: lustre-MDD0000: changelog off [ 6231.008292] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6231.023328] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6231.040765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6231.052309] Lustre: Skipped 5 previous similar messages [ 6231.538492] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6237.671610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6237.683418] Lustre: Skipped 4 previous similar messages [ 6242.794025] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6242.802844] Lustre: Skipped 1 previous similar message [ 6244.319133] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6244.608409] Lustre: server umount lustre-OST0000 complete [ 6253.025037] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6253.044938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6253.049079] Lustre: Skipped 1 previous similar message [ 6255.750490] Lustre: server umount lustre-MDT0000 complete [ 6258.146779] LustreError: 136018:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6258.180432] LustreError: 136018:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 6259.790228] LustreError: 136812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448410 with bad export cookie 11195723731662157686 [ 6259.792575] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6259.805801] LustreError: 136812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6260.236140] Lustre: server umount lustre-MDT0001 complete [ 6274.110933] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 6277.258441] Key type lgssc unregistered [ 6277.735491] LNet: 147129:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6277.751820] LNetError: 147129:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6277.775358] LNet: Removed LNI 192.168.206.151@tcp [ 6278.832318] Key type .llcrypt unregistered [ 6278.835858] Key type ._llcrypt unregistered [ 6300.856646] Key type ._llcrypt registered [ 6300.861682] Key type .llcrypt registered [ 6300.983434] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 6317.399615] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 6318.330024] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6318.337677] alg: No test for adler32 (adler32-zlib) [ 6319.367687] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6319.571211] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 6321.216324] Key type lgssc registered [ 6322.486248] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6356.882422] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6356.930238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6358.159592] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6358.217924] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6358.310485] Lustre: lustre-MDT0000: new disk, initializing [ 6358.365714] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6358.394565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6362.960876] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6372.385677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6372.476699] Lustre: 150334:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6372.542808] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6372.546486] Lustre: Skipped 1 previous similar message [ 6372.620973] Lustre: lustre-MDT0001: new disk, initializing [ 6372.744918] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6372.769521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6372.782228] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6377.114477] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6385.462739] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6390.825979] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6396.940699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6397.224554] Lustre: lustre-OST0000: new disk, initializing [ 6397.231455] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6397.239924] Lustre: 152300:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6397.341969] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6402.999143] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6406.768278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6406.784444] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6406.940271] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6413.150232] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6418.932104] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_module obdecho/obdecho [ 6420.861380] LustreError: 153303:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6420.871391] LustreError: 153303:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6422.585514] LustreError: 153400:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6434.279480] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6434.295661] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6434.313771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6437.859978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6437.865444] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6437.887108] Lustre: Skipped 1 previous similar message [ 6440.170672] Lustre: server umount lustre-MDT0000 complete [ 6443.601160] LustreError: 150326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448594 with bad export cookie 16113038244640735510 [ 6443.617426] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6443.623222] LustreError: 150326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6443.937239] Lustre: server umount lustre-MDT0001 complete [ 6457.346789] Lustre: server umount lustre-OST0000 complete [ 6482.419930] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 6495.512595] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 6533.679409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6534.018867] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6534.069561] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6534.155859] Lustre: lustre-MDT0000: new disk, initializing [ 6534.226522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6534.252354] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6539.715992] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6552.079979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6552.243642] Lustre: 157116:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6552.294648] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6552.301703] Lustre: Skipped 1 previous similar message [ 6552.373515] Lustre: lustre-MDT0001: new disk, initializing [ 6552.445081] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6552.484195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6552.496742] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6557.942655] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6568.099259] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6574.065382] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6580.582058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6580.858557] Lustre: lustre-OST0000: new disk, initializing [ 6580.861940] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6580.868953] Lustre: 159085:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6580.950716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6586.434470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6586.444661] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6586.548556] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6586.794320] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6596.709416] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6601.347720] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6601.528560] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6606.544542] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6606.780445] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6612.450402] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6612.453952] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6612.464343] LustreError: Skipped 1 previous similar message [ 6612.468508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6612.481013] Lustre: Skipped 1 previous similar message [ 6612.511454] Lustre: Skipped 3 previous similar messages [ 6617.593978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6617.601447] Lustre: Skipped 1 previous similar message [ 6622.694888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6622.714439] Lustre: Skipped 1 previous similar message [ 6623.199161] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6623.352925] Lustre: server umount lustre-OST0000 complete [ 6629.349250] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6629.360703] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6632.928829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6632.940120] Lustre: Skipped 1 previous similar message [ 6634.974425] Lustre: server umount lustre-MDT0000 complete [ 6638.049757] LustreError: 157123:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6638.075746] LustreError: 157123:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6638.607663] LustreError: 157108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448789 with bad export cookie 16113038244640736476 [ 6638.608579] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6638.620454] LustreError: 157108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6639.265213] Lustre: server umount lustre-MDT0001 complete [ 6643.193898] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 11:19:51 (1788448791) [ 6645.226816] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6647.425646] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 11:19:56 (1788448796) [ 6656.547782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6657.516478] LustreError: 161500:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6657.650401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6663.136965] LustreError: 161501:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6664.070506] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6667.239329] LustreError: 161500:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6672.381693] LustreError: 161501:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6674.535963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6679.566358] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6688.572057] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6694.147478] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6701.129376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6701.549128] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6701.554493] Lustre: Skipped 1 previous similar message [ 6707.953489] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6715.733560] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6727.476299] Lustre: server umount lustre-MDT0000 complete [ 6730.211096] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6730.222155] Lustre: Skipped 1 previous similar message [ 6730.225203] LustreError: 161497:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6730.257117] LustreError: 161497:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6730.846605] LustreError: 161481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448881 with bad export cookie 16113038244640737358 [ 6730.854889] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6730.859140] LustreError: 161481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6731.770726] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.51@tcp (stopping) [ 6731.781710] Lustre: Skipped 1 previous similar message [ 6737.321378] Lustre: server umount lustre-MDT0001 complete [ 6743.956138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6757.739563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6758.049976] LustreError: 165343:0:(llog.c:1655:llog_backup()) MGC192.168.206.151@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6758.053966] Lustre: 165343:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.151@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6760.928455] LustreError: 165348:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6760.953992] LustreError: 165348:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6793.166816] LustreError: 165350:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6793.187545] LustreError: 165350:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 24 previous similar messages [ 6818.783554] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6818.790780] Lustre: 165343:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.206.151@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6858.690513] LustreError: 165370:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6858.711111] LustreError: 165370:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 42 previous similar messages [ 6859.778855] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6863.925876] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6864.877281] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6872.115866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6880.857953] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6881.772729] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6888.440234] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6890.169932] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6896.147748] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6897.847877] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6917.601918] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6917.620223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6917.629784] Lustre: Skipped 3 previous similar messages [ 6917.651851] Lustre: Skipped 4 previous similar messages [ 6923.305423] Lustre: server umount lustre-MDT0000 complete [ 6927.134765] LustreError: 161482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449077 with bad export cookie 16113038244640743812 [ 6927.152290] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6927.477793] Lustre: server umount lustre-MDT0001 complete [ 6941.991667] Lustre: server umount lustre-OST0000 complete [ 6964.041072] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 6977.054521] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 7017.126536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7017.424065] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7017.447436] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7017.523858] Lustre: lustre-MDT0000: new disk, initializing [ 7017.601956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7017.608962] Lustre: Skipped 1 previous similar message [ 7017.628918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7022.041827] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7031.748635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7031.866252] Lustre: 170449:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7032.118879] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7036.088706] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7045.098309] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7049.843606] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7055.566469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7055.804700] Lustre: lustre-OST0000: new disk, initializing [ 7055.808124] Lustre: Skipped 1 previous similar message [ 7055.812388] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7055.817298] Lustre: Skipped 2 previous similar messages [ 7055.830965] Lustre: 172417:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7061.539307] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7066.154609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7066.164503] Lustre: Skipped 1 previous similar message [ 7066.173788] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7066.324676] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7071.234657] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7076.462357] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7076.742717] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7081.232796] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7081.413444] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7087.075510] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7087.076796] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7087.087112] LustreError: Skipped 1 previous similar message [ 7087.090903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7087.090913] Lustre: Skipped 3 previous similar messages [ 7087.145574] Lustre: Skipped 2 previous similar messages [ 7089.704798] Lustre: server umount lustre-OST0000 complete [ 7093.728671] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7099.535553] Lustre: server umount lustre-MDT0000 complete [ 7102.436800] LustreError: 171369:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7102.455170] LustreError: 171369:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [ 7103.065810] LustreError: 170441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449253 with bad export cookie 16113038244640760647 [ 7103.066359] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7103.079382] LustreError: 170441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7103.314264] Lustre: server umount lustre-MDT0001 complete [ 7111.521146] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 11:27:40 (1788449260) [ 7118.491786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7122.902976] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7130.795416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7134.737440] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7142.164969] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7150.330254] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7159.196719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7165.837881] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7172.195959] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7184.349918] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 11:28:53 (1788449333) [ 7189.472798] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7189.488728] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7189.498258] Lustre: Skipped 2 previous similar messages [ 7194.072111] Lustre: server umount lustre-OST0000 complete [ 7203.473136] Lustre: server umount lustre-MDT0000 complete [ 7206.543316] LustreError: 174810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449357 with bad export cookie 16113038244640761529 [ 7206.545883] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7206.550907] LustreError: 174810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7206.830755] Lustre: server umount lustre-MDT0001 complete [ 7216.603676] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 7219.181436] Key type lgssc unregistered [ 7219.464939] LNet: 178557:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7219.474582] LNetError: 178557:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7219.503960] LNet: Removed LNI 192.168.206.151@tcp [ 7220.370632] Key type .llcrypt unregistered [ 7220.375458] Key type ._llcrypt unregistered [ 7236.936905] Key type ._llcrypt registered [ 7236.942649] Key type .llcrypt registered [ 7237.109169] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 7238.263506] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7238.281795] alg: No test for adler32 (adler32-zlib) [ 7239.433629] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7239.676773] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 7241.335372] Key type lgssc registered [ 7242.798814] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7252.490678] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7252.535185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7268.127567] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7273.247442] LustreError: 179801:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.151@tcp: failed processing log, type 4: rc = -110 [ 7278.368754] LustreError: 179801:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7278.381955] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7278.389456] LustreError: MGC192.168.206.151@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7278.405772] LustreError: 179801:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7278.441245] LustreError: 179801:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7278.462694] LustreError: 179801:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7278.570563] Lustre: server umount lustre-OST0000 complete [ 7278.576689] LustreError: 179801:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7287.151033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7288.849715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7293.383881] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7299.042834] LustreError: 180300:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7299.081109] LustreError: 180300:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7301.903650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7302.158969] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7306.789610] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7313.060153] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7318.668840] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7325.447034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7325.699696] LustreError: 182119:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7325.714523] Lustre: 182119:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.206.151@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7325.883579] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7333.947290] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7345.692902] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7349.738954] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7349.765518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7349.768231] Lustre: Skipped 1 previous similar message [ 7349.791482] Lustre: Skipped 2 previous similar messages [ 7354.485444] Lustre: server umount lustre-MDT0000 complete [ 7354.849357] LustreError: 180300:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7354.875621] LustreError: 180300:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7357.835283] LustreError: 180281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449508 with bad export cookie 13705545801419756874 [ 7357.838095] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7357.843976] LustreError: 180281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7358.208478] Lustre: server umount lustre-MDT0001 complete [ 7372.043810] Lustre: server umount lustre-OST0000 complete [ 7381.508920] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 7390.552883] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 7427.034408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7427.235804] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7427.261186] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7427.331545] Lustre: lustre-MDT0000: new disk, initializing [ 7427.388422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7427.401719] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7432.018536] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7442.221178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7442.342831] Lustre: 185642:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7442.381479] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7442.386314] Lustre: Skipped 1 previous similar message [ 7442.440231] Lustre: lustre-MDT0001: new disk, initializing [ 7442.514042] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7442.559974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7442.576216] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7448.401554] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7461.066865] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7469.343967] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7477.068278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7477.295724] Lustre: lustre-OST0000: new disk, initializing [ 7477.299117] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7477.303740] Lustre: 187612:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7477.371991] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7482.555433] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7485.006386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7485.011705] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7485.119291] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7491.563418] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7495.731274] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7495.914239] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7500.134788] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7500.316769] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7505.889215] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7505.889276] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7505.902220] LustreError: Skipped 1 previous similar message [ 7505.903080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7505.943761] Lustre: Skipped 2 previous similar messages [ 7508.423374] Lustre: server umount lustre-OST0000 complete [ 7514.084782] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7514.100133] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7514.122848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7514.130828] Lustre: Skipped 1 previous similar message [ 7516.128216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7516.133607] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7521.250208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7521.259763] Lustre: Skipped 2 previous similar messages [ 7526.367161] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7526.598440] Lustre: server umount lustre-MDT0000 complete [ 7530.469125] LustreError: 185651:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7530.482493] LustreError: 185651:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7530.699572] LustreError: 185637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449681 with bad export cookie 13705545801419757595 [ 7530.700219] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7530.709033] LustreError: 185637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7531.156325] Lustre: server umount lustre-MDT0001 complete [ 7540.323176] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 11:34:49 (1788449689) [ 7558.241927] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 7560.591888] Key type lgssc unregistered [ 7560.880620] LNet: 190559:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7560.886725] LNetError: 190559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7560.907550] LNet: Removed LNI 192.168.206.151@tcp [ 7561.656463] Key type .llcrypt unregistered [ 7561.658079] Key type ._llcrypt unregistered [ 7578.658906] Key type ._llcrypt registered [ 7578.661890] Key type .llcrypt registered [ 7578.794896] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 7579.602325] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7579.636444] alg: No test for adler32 (adler32-zlib) [ 7580.718501] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7580.972908] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 7582.615184] Key type lgssc registered [ 7583.730834] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7590.108788] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7590.120632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7591.572585] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7595.506245] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7602.149891] LustreError: 191801:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7602.178965] LustreError: 191801:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7604.816897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7605.568499] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7611.492821] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7620.557578] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7627.285077] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7635.807922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7636.067952] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7636.073181] LustreError: 193668:0:(llog.c:1696:llog_backup()) MGC192.168.206.151@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7636.083935] Lustre: 193668:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.206.151@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7636.241111] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7643.432032] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7650.765616] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7655.908885] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7655.917012] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7655.933389] Lustre: Skipped 2 previous similar messages [ 7655.942150] Lustre: Skipped 1 previous similar message [ 7659.031239] Lustre: server umount lustre-MDT0000 complete [ 7661.040770] LustreError: 191802:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7661.062453] LustreError: 191802:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7662.633371] LustreError: 191781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449813 with bad export cookie 2427539783472957918 [ 7662.648794] LustreError: 191781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7662.655654] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7662.903595] Lustre: server umount lustre-MDT0001 complete [ 7675.418610] Lustre: server umount lustre-OST0000 complete [ 7684.097445] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 7694.342361] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 7732.745480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7733.261731] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7733.334125] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7733.415335] Lustre: lustre-MDT0000: new disk, initializing [ 7733.501361] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7733.525047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7739.566768] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7752.572257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7752.673242] Lustre: 197192:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7752.756130] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7752.758458] Lustre: Skipped 1 previous similar message [ 7752.762936] LustreError: 197975:0:(llog.c:1696:llog_backup()) MGC192.168.206.151@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7752.768805] LustreError: 197975:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7752.773874] Lustre: 197975:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.151@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7752.813773] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7752.821545] Lustre: Skipped 1 previous similar message [ 7752.969802] Lustre: lustre-MDT0001: new disk, initializing [ 7753.103682] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7753.157867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7753.181580] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7758.010753] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7768.521254] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7775.550980] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7783.571539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7783.795368] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7783.798060] Lustre: Skipped 1 previous similar message [ 7783.800926] LustreError: 199137:0:(llog.c:1696:llog_backup()) MGC192.168.206.151@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7783.808257] LustreError: 199137:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7783.813974] Lustre: 199137:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.151@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7783.825604] Lustre: 199137:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7783.843538] Lustre: lustre-OST0000: new disk, initializing [ 7783.847424] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7783.852701] Lustre: 199162:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7783.930339] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7790.692682] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7794.260109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7794.276746] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7794.434057] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7801.419315] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7806.505217] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7806.748993] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7811.535559] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7811.791679] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7815.140147] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7815.147965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7815.167787] Lustre: Skipped 1 previous similar message [ 7815.185841] Lustre: Skipped 2 previous similar messages [ 7820.260910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7820.268041] Lustre: Skipped 1 previous similar message [ 7825.378468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7825.391613] Lustre: Skipped 1 previous similar message [ 7827.937407] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7828.059212] Lustre: server umount lustre-OST0000 complete [ 7835.104205] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7835.116122] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7835.133958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7838.540465] Lustre: server umount lustre-MDT0000 complete [ 7840.739341] LustreError: 197199:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7840.751744] LustreError: 197199:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7842.364774] LustreError: 199161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449993 with bad export cookie 2427539783472958632 [ 7842.368897] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7842.384157] LustreError: 199161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7842.750463] Lustre: server umount lustre-MDT0001 complete [ 7853.456509] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 11:40:01 (1788450001) [ 7875.369421] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 7885.740253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7886.294678] LustreError: 202311:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7886.373854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7891.432801] LustreError: 202312:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7891.522750] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7901.667024] LustreError: 202311:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7901.683700] LustreError: 202311:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7901.898809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7906.693316] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7909.564364] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7914.996777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7922.011550] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7931.394761] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7931.645923] Lustre: lustre-OST0001: new disk, initializing [ 7931.651874] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7931.658078] Lustre: 204620:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7931.783577] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7931.791784] Lustre: Skipped 2 previous similar messages [ 7938.595705] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7939.173130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7939.186676] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7939.308640] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7949.657543] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7955.643832] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7957.830746] Lustre: DEBUG MARKER: ADD OST3 [ 7963.308189] loop: module loaded [ 7963.369316] loop0: detected capacity change from 0 to 204800000 [ 7963.655723] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7963.666770] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7963.691485] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7969.717159] loop0: detected capacity change from 0 to 204800000 [ 7970.909538] Lustre: DEBUG MARKER: STOP OST3 [ 7975.924360] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7976.094254] Lustre: lustre-OST0003: new disk, initializing [ 7976.097186] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7976.106588] Lustre: 206626:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7977.466045] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7977.630676] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7977.640631] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7977.686924] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7979.487521] LustreError: 206790:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7979.551431] Lustre: Failing over lustre-OST0003 [ 7979.791538] Lustre: server umount lustre-OST0003 complete [ 7982.855995] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7983.079170] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7983.085271] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7983.092562] Lustre: Skipped 1 previous similar message [ 7983.095723] LustreError: 205009:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7991.067866] Lustre: DEBUG MARKER: START OST3 again [ 7998.380827] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7998.689876] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7998.695476] Lustre: Skipped 1 previous similar message [ 7998.709775] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7998.927542] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8000.739244] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8000.742045] Lustre: lustre-OST0003: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 8005.180029] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8025.213322] Lustre: Failing over lustre-OST0003 [ 8025.310990] Lustre: server umount lustre-OST0003 complete [ 8026.593638] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8026.605249] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8026.618634] Lustre: Skipped 2 previous similar messages [ 8026.619777] LustreError: 203801:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8026.648172] LustreError: 203801:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 8045.024844] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8045.051875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8045.055121] Lustre: Skipped 3 previous similar messages [ 8048.574317] Lustre: server umount lustre-MDT0000 complete [ 8052.449537] LustreError: 202294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450203 with bad export cookie 2427539783472959521 [ 8052.456054] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8052.472688] LustreError: 202294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8058.938796] Lustre: server umount lustre-MDT0001 complete [ 8065.072632] Lustre: server umount lustre-OST0000 complete [ 8069.922278] Lustre: server umount lustre-OST0001 complete [ 8077.444299] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 8086.020973] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 8125.544878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8125.756766] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8125.775322] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8125.854752] Lustre: lustre-MDT0000: new disk, initializing [ 8125.933188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8130.628128] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8142.684536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8142.923539] Lustre: 212090:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8143.023227] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8143.033380] Lustre: Skipped 1 previous similar message [ 8143.178421] Lustre: lustre-MDT0001: new disk, initializing [ 8143.324883] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8143.340538] Lustre: Skipped 1 previous similar message [ 8143.406681] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8143.428808] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8148.325877] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8159.186239] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8166.247090] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8173.620412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8173.845535] Lustre: 214062:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8175.222948] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8175.392371] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8180.970796] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8189.902305] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8194.639552] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8194.789667] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8198.789978] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8198.972679] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8201.187908] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8201.192443] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8201.202847] LustreError: Skipped 2 previous similar messages [ 8201.207407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8201.221183] Lustre: Skipped 6 previous similar messages [ 8201.232459] Lustre: Skipped 5 previous similar messages [ 8206.639944] Lustre: server umount lustre-OST0000 complete [ 8216.796671] Lustre: server umount lustre-MDT0000 complete [ 8220.600566] LustreError: 212083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450371 with bad export cookie 2427539783472961817 [ 8220.607803] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8220.613507] LustreError: 212083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8220.920800] Lustre: server umount lustre-MDT0001 complete [ 8224.619096] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 11:46:13 (1788450373) [ 8240.312807] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 8250.291672] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 8287.301053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8287.537783] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8287.560910] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8287.565476] Lustre: Skipped 1 previous similar message [ 8287.667422] Lustre: lustre-MDT0000: new disk, initializing [ 8287.669732] Lustre: Skipped 1 previous similar message [ 8287.813208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8287.829791] Lustre: Skipped 1 previous similar message [ 8292.313562] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8302.899523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8303.127875] Lustre: 218671:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8303.508722] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8308.038328] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8317.629563] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8324.678469] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8332.334597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8332.541428] Lustre: 220639:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8333.950751] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8339.236962] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8349.204513] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8353.836631] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8354.065379] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8358.156533] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8358.447838] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8364.516890] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8364.518379] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8364.525528] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8364.525536] Lustre: Skipped 6 previous similar messages [ 8364.537023] LustreError: Skipped 1 previous similar message [ 8364.545917] Lustre: Skipped 3 previous similar messages [ 8366.708731] Lustre: server umount lustre-OST0000 complete [ 8377.018399] Lustre: server umount lustre-MDT0000 complete [ 8379.876489] LustreError: 218682:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8379.903266] LustreError: 218682:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 17 previous similar messages [ 8381.262914] LustreError: 219465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450532 with bad export cookie 2427539783472962699 [ 8381.266202] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8381.273706] LustreError: 219465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8381.653432] Lustre: server umount lustre-MDT0001 complete [ 8391.150695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8397.517147] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8407.800550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8408.311702] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8408.320271] Lustre: Skipped 5 previous similar messages [ 8413.295325] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8420.990413] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8427.242623] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8434.110516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8440.516923] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8447.145572] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8457.189360] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8457.206878] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8457.225219] Lustre: Skipped 2 previous similar messages [ 8457.232484] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8457.239565] Lustre: Skipped 3 previous similar messages [ 8463.125986] Lustre: server umount lustre-OST0000 complete [ 8481.248063] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8481.456768] Lustre: server umount lustre-MDT0000 complete [ 8485.722880] LustreError: 222838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450636 with bad export cookie 2427539783472963581 [ 8485.723971] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8485.737142] LustreError: 222838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8486.210767] Lustre: server umount lustre-MDT0001 complete [ 8584.628663] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 8588.008872] Key type lgssc unregistered [ 8588.508721] LNet: 226525:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8588.521887] LNetError: 226525:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8588.545789] LNet: Removed LNI 192.168.206.151@tcp [ 8589.790344] Key type .llcrypt unregistered [ 8589.798394] Key type ._llcrypt unregistered [ 8615.388355] Key type ._llcrypt registered [ 8615.390446] Key type .llcrypt registered [ 8615.493434] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 8631.868520] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 8632.943400] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8633.034428] alg: No test for adler32 (adler32-zlib) [ 8634.069976] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 8634.335561] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 8636.079511] Key type lgssc registered [ 8637.544746] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8677.220271] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8677.275529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8678.523856] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8678.582347] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8678.692777] Lustre: lustre-MDT0000: new disk, initializing [ 8678.805478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8678.840607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8684.030451] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8695.587984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8695.726181] Lustre: 229879:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8695.791306] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8695.796225] Lustre: Skipped 1 previous similar message [ 8695.914486] Lustre: lustre-MDT0001: new disk, initializing [ 8696.043520] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8696.095431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8696.117989] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8701.128207] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8711.446593] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8717.589962] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8724.595666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8724.930943] Lustre: lustre-OST0000: new disk, initializing [ 8724.939823] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8724.946232] Lustre: 231847:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8725.035337] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8731.300225] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8732.745412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8732.759182] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8732.895833] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8741.587464] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8746.719776] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8746.924780] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8751.969498] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8752.205270] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8758.758261] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8758.760799] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8758.776107] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8758.779809] Lustre: Skipped 1 previous similar message [ 8758.789436] Lustre: Skipped 1 previous similar message [ 8759.993492] Lustre: server umount lustre-OST0000 complete [ 8767.970963] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8767.979435] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8767.997038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8769.002170] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8769.035337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8769.049843] Lustre: Skipped 1 previous similar message [ 8774.112342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8774.115220] Lustre: Skipped 1 previous similar message [ 8778.211061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8778.221288] Lustre: Skipped 1 previous similar message [ 8779.231229] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8779.522647] Lustre: server umount lustre-MDT0000 complete [ 8783.424597] LustreError: 229870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450934 with bad export cookie 14892816819443779 [ 8783.433170] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8783.444228] LustreError: 229870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8783.846422] Lustre: server umount lustre-MDT0001 complete [ 8803.562412] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 8814.903430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8815.111988] LustreError: 234780:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8815.124490] LustreError: 234780:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8815.153441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8819.951609] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8820.192494] LustreError: 234781:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8824.298828] LustreError: 234780:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8829.418359] LustreError: 234781:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8830.962422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8831.658977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8837.639992] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8841.376834] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8849.153989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8849.496924] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8857.175248] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8867.660184] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8867.915473] Lustre: lustre-OST0001: new disk, initializing [ 8867.923807] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8867.943341] Lustre: 237088:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8868.093861] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8874.429227] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8877.164814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8877.177114] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8877.278530] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8886.279243] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8894.499472] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8910.651505] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 11:57:38 (1788451058) [ 8913.381120] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8913.382058] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8913.392427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8913.431585] Lustre: Skipped 3 previous similar messages [ 8919.688915] Lustre: server umount lustre-MDT0000 complete [ 8923.621723] LustreError: 234781:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8923.639464] LustreError: 234781:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8923.701959] LustreError: 234761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451074 with bad export cookie 14892816819444661 [ 8923.709396] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8923.715512] LustreError: 234761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8924.086959] Lustre: server umount lustre-MDT0001 complete [ 8939.114366] Lustre: server umount lustre-OST0000 complete [ 8952.550946] Lustre: server umount lustre-OST0001 complete [ 8960.374139] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 8971.121702] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 9015.627638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9015.950327] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9016.001416] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9016.163641] Lustre: lustre-MDT0000: new disk, initializing [ 9016.315604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9016.344233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9022.035593] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9032.588957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9032.752679] Lustre: 241038:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9032.826847] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9032.831661] Lustre: Skipped 1 previous similar message [ 9032.932544] Lustre: lustre-MDT0001: new disk, initializing [ 9033.092168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9033.105304] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9038.233784] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9049.730754] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9055.096390] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9062.224522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9062.630666] Lustre: lustre-OST0000: new disk, initializing [ 9062.635474] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9062.642644] Lustre: 243008:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9063.933208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9063.946474] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9064.213607] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9069.304708] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9079.556734] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9085.008724] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9085.202897] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9089.546936] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9089.726287] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9094.624326] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9094.631963] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9094.656026] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9094.659524] Lustre: Skipped 7 previous similar messages [ 9097.832507] Lustre: server umount lustre-OST0000 complete [ 9104.864688] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9104.873452] LustreError: Skipped 1 previous similar message [ 9104.875526] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9104.906591] Lustre: Skipped 1 previous similar message [ 9108.324499] Lustre: server umount lustre-MDT0000 complete [ 9110.511125] LustreError: 242169:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9110.547192] LustreError: 242169:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9112.240700] LustreError: 241028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451263 with bad export cookie 14892816819446796 [ 9112.243826] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9112.251521] LustreError: 241028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9112.654168] Lustre: server umount lustre-MDT0001 complete [ 9121.282892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9121.774361] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9121.781708] Lustre: Skipped 2 previous similar messages [ 9126.138181] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9126.882285] LustreError: 245233:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9126.907497] LustreError: 245233:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9134.823283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9139.593817] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9147.542972] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9152.869950] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9159.813179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9166.241667] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9174.567386] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9325.031210] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9325.035344] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9325.051080] LustreError: Skipped 1 previous similar message [ 9325.051951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9325.064851] Lustre: Skipped 2 previous similar messages [ 9325.070551] Lustre: Skipped 5 previous similar messages [ 9327.436274] Lustre: server umount lustre-OST0000 complete [ 9338.291875] Lustre: server umount lustre-MDT0000 complete [ 9340.388577] LustreError: 245960:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9340.414831] LustreError: 245960:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9342.266881] LustreError: 245213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451493 with bad export cookie 14892816819447678 [ 9342.270911] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9342.284291] LustreError: 245213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9342.741914] Lustre: server umount lustre-MDT0001 complete [ 9443.888456] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 9446.500309] Key type lgssc unregistered [ 9446.755708] LNet: 248847:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9446.760195] LNetError: 248847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9446.779386] LNet: Removed LNI 192.168.206.151@tcp [ 9447.738152] Key type .llcrypt unregistered [ 9447.748928] Key type ._llcrypt unregistered [ 9471.436388] Key type ._llcrypt registered [ 9471.441859] Key type .llcrypt registered [ 9471.589117] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 9486.925534] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 9488.096745] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9488.151474] alg: No test for adler32 (adler32-zlib) [ 9489.364269] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 9489.630451] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 9491.336518] Key type lgssc registered [ 9492.456608] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9538.176570] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 9549.210723] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9549.249390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9550.461257] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9550.489284] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9550.618159] Lustre: lustre-MDT0000: new disk, initializing [ 9550.747223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9550.780834] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9555.101587] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9565.106016] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9565.230692] Lustre: 252921:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9565.283855] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9565.289266] Lustre: Skipped 1 previous similar message [ 9565.345544] Lustre: lustre-MDT0001: new disk, initializing [ 9565.434253] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9565.465304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9565.482134] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9569.835616] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9574.508381] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9580.827920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9581.181813] Lustre: lustre-OST0000: new disk, initializing [ 9581.189127] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9581.200543] Lustre: 254550:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9581.307298] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9582.401861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9582.414440] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9582.594291] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9588.780166] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9599.964493] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9600.233398] Lustre: lustre-OST0001: new disk, initializing [ 9600.239487] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9600.250457] Lustre: 255422:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9600.366227] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9606.806906] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9609.307260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9609.317336] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9609.418491] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9617.911608] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9627.074691] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9640.424518] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9640.426301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9640.450277] Lustre: Skipped 2 previous similar messages [ 9640.464844] Lustre: Skipped 3 previous similar messages [ 9645.029422] Lustre: server umount lustre-MDT0000 complete [ 9648.569906] LustreError: 252912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451799 with bad export cookie 5750817864863203968 [ 9648.578382] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9648.584035] LustreError: 252912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9648.866179] Lustre: server umount lustre-MDT0001 complete [ 9663.135569] Lustre: server umount lustre-OST0000 complete [ 9667.021505] Lustre: 250426:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788451801/real 1788451801] req@ffff90eb4fee8a80 x1875327466943616/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788451817 ref 1 fl Rpc:EXNQr/200/ffffffff rc -5/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9676.444861] Lustre: server umount lustre-OST0001 complete [ 9690.491971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9699.761822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9708.982228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9719.293230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9740.270798] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 9752.705529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9752.724844] Lustre: lustre-MDT0000: reset Object Index mappings [ 9753.270198] LustreError: 260435:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9753.299063] LustreError: 260435:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 9753.403364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9757.754949] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9758.688712] LustreError: 260436:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9763.816267] LustreError: 260435:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9766.141630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9766.164446] Lustre: lustre-MDT0001: reset Object Index mappings [ 9766.725936] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9771.145134] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9774.249662] Lustre: 261576:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9780.502103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9788.010096] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9792.041796] LustreError: 261932:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9796.486972] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9796.719985] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9796.727881] Lustre: Skipped 1 previous similar message [ 9799.817634] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9799.835225] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9801.975616] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9808.802046] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9812.054459] Lustre: 263451:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9818.083209] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9818.088448] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9818.107495] Lustre: Skipped 1 previous similar message [ 9818.111637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9819.448831] Lustre: server umount lustre-MDT0000 complete [ 9820.647394] LustreError: 260436:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9820.660388] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9820.691029] Lustre: Skipped 1 previous similar message [ 9823.351016] LustreError: 260416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451974 with bad export cookie 5750817864863210758 [ 9823.354059] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9823.365507] LustreError: 260416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9823.737074] Lustre: server umount lustre-MDT0001 complete [ 9837.969878] Lustre: server umount lustre-OST0000 complete [ 9851.993615] Lustre: server umount lustre-OST0001 complete [ 9869.415380] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 12:13:38 (1788452018) [ 9885.678823] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 9894.995601] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 9928.296708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9928.489864] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9928.508808] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9928.579214] Lustre: lustre-MDT0000: new disk, initializing [ 9928.660439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9928.693343] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9932.434505] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9943.108825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9943.282814] Lustre: 267704:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9943.347715] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9943.352440] Lustre: Skipped 1 previous similar message [ 9943.470595] Lustre: lustre-MDT0001: new disk, initializing [ 9943.658372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9943.668313] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9948.099700] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9956.298000] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9962.048838] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9969.064195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9969.513951] Lustre: lustre-OST0000: new disk, initializing [ 9969.518331] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9969.525614] Lustre: 269672:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9971.065357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9971.072568] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9971.177640] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9975.402230] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9984.089851] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9988.638852] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9988.838596] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9993.113460] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9993.433317] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9996.769725] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9996.775028] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9996.783210] Lustre: Skipped 1 previous similar message [ 9996.788940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10001.416891] Lustre: server umount lustre-OST0000 complete [10007.020877] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10007.047270] Lustre: Skipped 1 previous similar message [10007.062142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10007.079295] Lustre: Skipped 2 previous similar messages [10011.998281] Lustre: server umount lustre-MDT0000 complete [10012.128382] LustreError: 268515:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10012.141084] LustreError: 268515:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [10016.161632] LustreError: 267696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452166 with bad export cookie 5750817864863212851 [10016.169626] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10016.178517] LustreError: 267696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10016.669890] Lustre: server umount lustre-MDT0001 complete [10035.919791] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [10045.073586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10045.456418] LustreError: 272610:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10045.508436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10045.516111] Lustre: Skipped 2 previous similar messages [10049.695443] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10057.874551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10061.626153] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10064.480752] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10070.533684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10076.422956] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10083.425381] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10083.591581] Lustre: lustre-OST0001: new disk, initializing [10083.594851] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10083.599611] Lustre: 274915:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10084.881809] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10084.887695] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10085.010597] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10089.183232] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10099.041502] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10103.582833] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10109.734505] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10109.770910] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10117.092265] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10117.105031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10117.117789] Lustre: Skipped 4 previous similar messages [10117.127944] Lustre: Skipped 3 previous similar messages [10122.510036] Lustre: server umount lustre-MDT0000 complete [10126.935652] LustreError: 272591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452277 with bad export cookie 5750817864863213733 [10126.941463] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10126.955959] LustreError: 272591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10127.333995] Lustre: server umount lustre-MDT0001 complete [10141.759447] Lustre: server umount lustre-OST0000 complete [10143.711864] Lustre: 250424:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788452278/real 1788452278] req@ffff90eb492e3480 x1875327467092352/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788452294 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10143.728652] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10145.179057] Lustre: server umount lustre-OST0001 complete [10159.801810] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [10168.565830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10168.957921] LustreError: 277968:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10168.971709] LustreError: 277968:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10172.443383] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10180.028745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10180.387032] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10180.396633] Lustre: Skipped 4 previous similar messages [10184.814632] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10187.858725] Lustre: 279108:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10193.273471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10194.586861] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10199.357709] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10207.254565] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10212.600175] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10212.867403] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10219.689763] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10223.276216] Lustre: 280986:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10228.292108] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10228.341730] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10236.282316] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 12:19:45 (1788452385) [10252.257975] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10252.267892] LustreError: Skipped 1 previous similar message [10252.270699] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10252.283039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10252.290546] Lustre: Skipped 4 previous similar messages [10256.064094] Lustre: server umount lustre-MDT0000 complete [10258.921832] LustreError: 277964:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10258.937479] LustreError: 277964:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [10259.078661] LustreError: 277948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452409 with bad export cookie 5750817864863220047 [10259.086906] LustreError: 277948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10259.089251] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10265.379101] Lustre: server umount lustre-MDT0001 complete [10274.724352] Lustre: server umount lustre-OST0000 complete [10283.516278] Lustre: server umount lustre-OST0001 complete [10289.427350] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [10297.304391] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [10332.324376] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [10340.487635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10340.694848] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10340.717408] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10340.772344] Lustre: lustre-MDT0000: new disk, initializing [10340.831595] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10344.750281] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10352.246136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10352.354830] Lustre: 285223:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10352.561794] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10355.445835] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10365.684623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10365.973804] Lustre: 286854:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10367.734408] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10372.771522] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10383.944513] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10384.109235] Lustre: 287726:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10385.991622] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10386.000927] Lustre: Skipped 1 previous similar message [10386.075954] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10389.646462] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10398.441306] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10401.498256] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10401.504591] Lustre: Skipped 1 previous similar message [10416.178234] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 12:22:45 (1788452565) [10419.169831] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10419.186844] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10419.215481] Lustre: Skipped 5 previous similar messages [10419.225162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10419.238394] Lustre: Skipped 6 previous similar messages [10420.768080] Lustre: server umount lustre-MDT0000 complete [10424.379433] LustreError: 285213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452575 with bad export cookie 5750817864863227957 [10424.385690] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10424.404366] LustreError: 285213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10424.777146] Lustre: server umount lustre-MDT0001 complete [10437.348069] Lustre: server umount lustre-OST0000 complete [10450.389768] Lustre: server umount lustre-OST0001 complete [10457.334434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10457.726365] LustreError: 289973:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10457.748990] LustreError: 289973:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10457.827774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10457.836992] Lustre: Skipped 6 previous similar messages [10461.078287] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10467.948946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10471.511859] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10477.697540] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10482.514543] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10488.260558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10493.724099] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10500.260326] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10518.569669] loop1: detected capacity change from 0 to 204800000 [10518.698505] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10518.707323] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10518.731402] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10523.297590] loop1: detected capacity change from 0 to 204800000 [10527.556733] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10527.615244] LustreError: 289964:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs27967: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10527.626869] LustreError: 289964:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs27967 [10527.633981] LustreError: 289964:0:(mgs_handler.c:517:mgs_target_reg()) fs27967-MDT0001: Error checking target index: rc = -13 [10527.642843] LustreError: fs27967-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10527.654825] LustreError: 293692:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10527.659669] LustreError: 293692:0:(tgt_mount.c:2129:server_put_super()) no obd fs27967-MDT0001 [10527.663627] LustreError: 293692:0:(tgt_mount.c:116:server_deregister_mount()) fs27967-MDT0001 not registered [10527.695021] Lustre: server umount fs27967-MDT0001 complete [10527.701068] LustreError: 293692:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [10534.867863] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10535.050146] Lustre: Setting parameter fs27967-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs27967-MDT0001 [10535.111900] Lustre: srv-fs27967-MDT0001: No data found on store. Initialize space. [10535.118895] Lustre: Skipped 4 previous similar messages [10535.229740] Lustre: fs27967-MDT0001: new disk, initializing [10535.232560] Lustre: Skipped 3 previous similar messages [10535.364214] Lustre: srv-fs27967-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10535.370776] Lustre: 294158:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs27967-MDT0001-mdtlov: can't create new llog: rc = -115 [10535.391369] LustreError: 294158:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs27967-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10540.366574] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10547.544128] Lustre: server umount fs27967-MDT0001 complete [10562.624703] loop1: detected capacity change from 0 to 204800000 [10567.885070] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10568.224174] Lustre: 295725:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs27967-MDT0001-mdtlov: can't create new llog: rc = -115 [10568.238534] LustreError: 295725:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs27967-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10572.266959] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10576.696425] Lustre: server umount fs27967-MDT0001 complete [10579.946440] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10586.080895] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10586.117065] Lustre: Skipped 3 previous similar messages [10586.127853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10586.135411] Lustre: Skipped 4 previous similar messages [10591.704819] Lustre: server umount lustre-MDT0000 complete [10601.857901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10602.103780] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10606.576965] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10607.602273] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10613.560075] loop1: detected capacity change from 0 to 204800000 [10618.777398] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10619.263717] Lustre: 298169:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs27967-MDT0001-mdtlov: can't create new llog: rc = -115 [10619.270846] LustreError: 298169:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs27967-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10623.254178] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10627.270963] Lustre: server umount fs27967-MDT0001 complete [10647.521800] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10647.534795] LustreError: Skipped 2 previous similar messages [10651.879727] Lustre: server umount lustre-OST0000 complete [10658.789150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10658.792668] Lustre: Skipped 8 previous similar messages [10661.783302] Lustre: server umount lustre-MDT0000 complete [10666.264786] LustreError: 290716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452817 with bad export cookie 5750817864863230638 [10666.274857] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10666.282616] LustreError: 290716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10667.159873] Lustre: server umount lustre-MDT0001 complete [10681.554980] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [10684.554773] Key type lgssc unregistered [10684.826369] LNet: 300515:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10684.834925] LNetError: 300515:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10685.864697] LNet: Removed LNI 192.168.206.151@tcp [10686.745916] Key type .llcrypt unregistered [10686.748370] Key type ._llcrypt unregistered [10707.346669] Key type ._llcrypt registered [10707.348366] Key type .llcrypt registered [10707.500249] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [10708.852776] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10708.896247] alg: No test for adler32 (adler32-zlib) [10709.882495] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [10710.137540] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [10711.855198] Key type lgssc registered [10713.000814] Lustre: Echo OBD driver; http://www.lustre.org/ [10719.645681] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10719.680597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10721.123852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10725.327221] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10730.468086] LustreError: 301787:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10730.493105] LustreError: 301787:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10733.837249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10734.162066] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10738.284753] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10744.235920] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10748.503668] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10753.914781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10754.414909] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10758.505705] LustreError: 303618:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10760.590619] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10763.762476] LustreError: 303620:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10765.818152] LustreError: 303619:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10765.829550] LustreError: 303619:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10769.654121] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10770.917808] LustreError: 303620:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10770.949877] LustreError: 303620:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10781.159905] LustreError: 303619:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10781.173970] LustreError: 303619:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10785.367487] loop1: detected capacity change from 0 to 204800000 [10785.452988] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10785.460980] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10785.471630] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10790.540376] loop1: detected capacity change from 0 to 204800000 [10795.683420] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10795.864293] Lustre: fs1463-OST0000: new disk, initializing [10795.868252] Lustre: srv-fs1463-OST0000: No data found on store. Initialize space. [10795.882530] Lustre: 305523:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10795.954336] Lustre: fs1463-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10800.815468] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10800.962806] LustreError: 303852:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10800.982244] LustreError: 303852:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [10808.359514] Lustre: server umount fs1463-OST0000 complete [10819.664199] loop1: detected capacity change from 0 to 204800000 [10825.972644] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10826.296068] Lustre: fs1463-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10831.854577] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10836.935369] LustreError: 303619:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10836.956958] LustreError: 303619:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 20 previous similar messages [10840.121802] loop2: detected capacity change from 0 to 204800000 [10840.302697] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10840.343494] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10840.367546] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10842.615291] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10845.859534] loop2: detected capacity change from 0 to 204800000 [10847.736110] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10851.365386] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10851.467731] LustreError: fs1463-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10851.487123] LustreError: fs1463-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10851.508097] LustreError: 308561:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10851.519980] LustreError: 308561:0:(tgt_mount.c:2129:server_put_super()) no obd fs1463-OST0001 [10851.531687] LustreError: 308561:0:(tgt_mount.c:116:server_deregister_mount()) fs1463-OST0001 not registered [10851.577228] Lustre: server umount fs1463-OST0001 complete [10851.580417] LustreError: 308561:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [10864.128793] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10864.277902] Lustre: fs1463-OST0001: new disk, initializing [10864.285293] Lustre: srv-fs1463-OST0001: No data found on store. Initialize space. [10864.296731] Lustre: 309113:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10864.387813] Lustre: fs1463-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10870.526708] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10878.314798] Lustre: server umount fs1463-OST0000 complete [10886.932842] Lustre: server umount fs1463-OST0001 complete [10902.314890] loop1: detected capacity change from 0 to 204800000 [10903.504206] LustreError: 303618:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10903.526319] LustreError: 303618:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 38 previous similar messages [10907.746373] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10907.892992] Lustre: fs1463-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10913.342107] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10919.483034] loop2: detected capacity change from 0 to 204800000 [10924.473144] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10931.004339] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [10939.232796] Lustre: server umount fs1463-OST0000 complete [10947.774797] Lustre: server umount fs1463-OST0001 complete [10960.363647] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10960.364548] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10960.392131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10960.400897] Lustre: Skipped 1 previous similar message [10965.172674] Lustre: server umount lustre-OST0000 complete [10969.578309] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10969.591451] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10969.610874] Lustre: Skipped 1 previous similar message [10969.621875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10970.595684] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10975.054991] Lustre: server umount lustre-MDT0000 complete [10978.524124] LustreError: 301768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453129 with bad export cookie 8713161346482202984 [10978.531763] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10978.541029] LustreError: 301768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10978.895412] Lustre: server umount lustre-MDT0001 complete [10988.726614] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [10991.404388] Key type lgssc unregistered [10991.661606] LNet: 315037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10991.672992] LNetError: 315037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10991.694901] LNet: Removed LNI 192.168.206.151@tcp [10992.495056] Key type .llcrypt unregistered [10992.497956] Key type ._llcrypt unregistered [11021.826332] Key type ._llcrypt registered [11021.831980] Key type .llcrypt registered [11022.001304] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [11042.424801] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [11044.255885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11044.370015] alg: No test for adler32 (adler32-zlib) [11045.499774] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11045.857416] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [11047.647193] Key type lgssc registered [11048.802506] Lustre: Echo OBD driver; http://www.lustre.org/ [11085.456673] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11085.480028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11086.665569] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11086.688807] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11086.862750] Lustre: lustre-MDT0000: new disk, initializing [11086.967151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11087.030507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11091.169978] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [11102.202056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11102.286748] Lustre: 318912:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11102.357659] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11102.360579] Lustre: Skipped 1 previous similar message [11102.488163] Lustre: lustre-MDT0001: new disk, initializing [11102.576432] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11102.609740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11102.617189] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11106.915953] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [11116.849907] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11122.943954] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11130.290535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11130.493912] Lustre: lustre-OST0000: new disk, initializing [11130.497829] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11130.504632] Lustre: 320883:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11130.583359] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11136.573253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11136.579904] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11136.714791] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11136.898276] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [11146.785601] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11148.669871] Lustre: Mounted lustre-client - version 2.17.56_51_g2aa7cea [11149.583168] Lustre: Unmounted lustre-client [11162.599264] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11162.601838] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11162.612786] LustreError: Skipped 1 previous similar message [11162.641409] Lustre: Skipped 1 previous similar message [11162.643671] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11162.663717] Lustre: Skipped 1 previous similar message [11164.130138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11164.138261] Lustre: Skipped 1 previous similar message [11165.337873] Lustre: server umount lustre-OST0000 complete [11169.248976] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11169.264384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11169.281645] Lustre: Skipped 1 previous similar message [11174.374210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11174.377838] Lustre: Skipped 1 previous similar message [11175.572694] Lustre: server umount lustre-MDT0000 complete [11180.310534] LustreError: 318903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453331 with bad export cookie 17001056363094096041 [11180.322952] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11180.341284] LustreError: 318903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11180.777354] Lustre: server umount lustre-MDT0001 complete [11193.617348] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [11196.551845] Key type lgssc unregistered [11201.567427] LNet: 1 local NIs in recovery (showing 1): 192.168.206.151@tcp [11202.080608] LNet: 323053:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11202.092589] LNetError: 323053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11277.862138] LNet: Removed LNI 192.168.206.151@tcp [11278.627966] Key type .llcrypt unregistered [11278.630263] Key type ._llcrypt unregistered [11301.400802] Key type ._llcrypt registered [11301.406078] Key type .llcrypt registered [11301.572641] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [11317.843355] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [11318.926734] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11319.087697] alg: No test for adler32 (adler32-zlib) [11320.263498] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11320.562741] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [11322.247169] Key type lgssc registered [11323.470939] Lustre: Echo OBD driver; http://www.lustre.org/ [11354.586800] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 12:38:23 (1788453503) [11361.115897] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11361.131076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11362.430561] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11362.454293] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11362.533528] Lustre: lustre-MDT0000: new disk, initializing [11362.627128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11362.655151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11367.282674] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [11377.955966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11378.078059] Lustre: 326275:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11378.161308] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11378.165716] Lustre: Skipped 1 previous similar message [11378.248856] Lustre: lustre-MDT0001: new disk, initializing [11378.342115] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11378.370414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11378.380955] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11382.724496] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [11392.337279] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11398.074882] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11404.993482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11405.225379] Lustre: lustre-OST0000: new disk, initializing [11405.228969] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11405.234580] Lustre: 328245:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11405.319650] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11411.022423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11411.041786] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11411.151509] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [11411.179960] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11420.209586] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11442.148082] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11442.164752] Lustre: Skipped 1 previous similar message [11442.168860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11442.169631] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11442.196702] Lustre: Skipped 1 previous similar message [11443.796334] Lustre: server umount lustre-OST0000 complete [11450.336477] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11450.350297] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11450.375070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11452.385858] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11452.388591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11452.411170] Lustre: Skipped 1 previous similar message [11454.845162] Lustre: server umount lustre-MDT0000 complete [11459.196536] LustreError: 328255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453610 with bad export cookie 9140845485137782988 [11459.201218] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11459.231108] LustreError: 328255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11459.970856] Lustre: server umount lustre-MDT0001 complete [11472.620592] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [11475.471983] Key type lgssc unregistered [11475.830401] LNet: 330155:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11475.840574] LNetError: 330155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11475.872977] LNet: Removed LNI 192.168.206.151@tcp [11476.869134] Key type .llcrypt unregistered [11476.872721] Key type ._llcrypt unregistered [11499.114707] Key type ._llcrypt registered [11499.124493] Key type .llcrypt registered [11499.326757] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [11500.615053] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11500.698135] alg: No test for adler32 (adler32-zlib) [11502.030416] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11502.327843] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [11503.960124] Key type lgssc registered [11505.221955] Lustre: Echo OBD driver; http://www.lustre.org/ [11513.064893] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11513.097348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11514.564796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11519.399647] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [11525.095402] LustreError: 331432:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11525.127153] LustreError: 331432:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11528.947474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11529.410856] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11535.114560] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [11543.375978] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11549.998301] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11557.582604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11557.946245] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11566.415832] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [11575.761489] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11585.916962] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11588.049861] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11593.691101] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.51@tcp (stopping) [11594.724583] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11594.735967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11594.754826] Lustre: Skipped 1 previous similar message [11594.762881] Lustre: Skipped 2 previous similar messages [11598.796374] LustreError: 333732:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11598.865933] Lustre: server umount lustre-MDT0000 complete [11599.843207] LustreError: 331428:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11599.876241] LustreError: 331428:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11602.409441] LustreError: 331413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453753 with bad export cookie 11129842608571134622 [11602.437570] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11603.913200] LustreError: 331429:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11603.954276] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.51@tcp (stopping) [11604.963328] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11604.980582] Lustre: Skipped 1 previous similar message [11607.986316] LustreError: 332733:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11607.987685] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11608.010018] LustreError: 332733:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11608.027024] Lustre: Skipped 1 previous similar message [11608.847214] Lustre: server umount lustre-MDT0001 complete [11612.580812] Lustre: server umount lustre-OST0000 complete [11620.468773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11625.176443] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [11628.227895] Lustre: 335611:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.151@tcp [11630.192661] Lustre: server umount MGS complete [11638.230512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11638.903563] LustreError: 336183:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11639.018634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11642.793348] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [11651.577792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11651.864023] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11656.245314] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [11665.362826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11665.715209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11671.540830] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [11680.413389] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11682.288351] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11689.292373] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11691.183362] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11697.130388] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [11697.137475] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11697.162313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11697.632147] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11699.297331] Lustre: server umount lustre-OST0000 complete [11704.808198] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11704.830242] Lustre: Skipped 2 previous similar messages [11709.921177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11709.937758] Lustre: Skipped 4 previous similar messages [11718.114707] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11718.406826] Lustre: server umount lustre-MDT0000 complete [11720.165107] LustreError: 337505:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11720.191172] LustreError: 337505:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [11721.781859] LustreError: 336163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453872 with bad export cookie 11129842608571135462 [11721.800222] LustreError: 336163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [11721.801836] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11722.152142] Lustre: server umount lustre-MDT0001 complete [11734.177750] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [11736.639448] Key type lgssc unregistered [11737.007543] LNet: 339625:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11737.014541] LNetError: 339625:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11737.041877] LNet: Removed LNI 192.168.206.151@tcp [11738.001151] Key type .llcrypt unregistered [11738.003134] Key type ._llcrypt unregistered [11761.946711] Key type ._llcrypt registered [11761.949160] Key type .llcrypt registered [11762.085748] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [11781.011993] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [11782.058147] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11782.111738] alg: No test for adler32 (adler32-zlib) [11783.332134] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11783.571796] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [11785.287656] Key type lgssc registered [11786.408232] Lustre: Echo OBD driver; http://www.lustre.org/ [11822.033753] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11822.051178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11823.351487] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11823.387248] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11823.482345] Lustre: lustre-MDT0000: new disk, initializing [11823.584793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11823.621315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11827.595231] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [11838.604526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11838.772678] Lustre: 342790:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11838.843983] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11838.848692] Lustre: Skipped 1 previous similar message [11838.937324] Lustre: lustre-MDT0001: new disk, initializing [11839.084721] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11839.114735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11839.130593] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11844.241897] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [11854.403204] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11860.473482] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11867.712145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11868.118563] Lustre: lustre-OST0000: new disk, initializing [11868.130332] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11868.142595] Lustre: 344757:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11868.245473] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11873.360396] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11873.370616] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11873.470473] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11874.916202] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [11886.219865] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11891.285738] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11891.535788] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11896.907892] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11897.144719] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11899.362296] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11899.375444] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11899.381449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11899.392080] Lustre: Skipped 1 previous similar message [11899.405448] Lustre: Skipped 1 previous similar message [11902.440162] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11902.449774] Lustre: Skipped 1 previous similar message [11905.454839] Lustre: server umount lustre-OST0000 complete [11910.624370] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11910.640941] LustreError: Skipped 1 previous similar message [11910.647948] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11910.673498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11912.676688] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11912.686735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11912.739403] Lustre: Skipped 1 previous similar message [11916.407289] Lustre: server umount lustre-MDT0000 complete [11920.682184] LustreError: 342783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454071 with bad export cookie 7988014769211226575 [11920.702993] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11920.714597] LustreError: 342783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11921.404817] Lustre: server umount lustre-MDT0001 complete [11932.604528] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 12:48:01 (1788454081) [11951.683826] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [11962.765499] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [11999.859248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12000.080559] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12000.106526] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12000.201419] Lustre: lustre-MDT0000: new disk, initializing [12000.273093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12000.309868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12005.101492] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [12016.222821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12016.372567] Lustre: 349524:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12016.470808] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12016.473605] Lustre: Skipped 1 previous similar message [12016.548920] Lustre: lustre-MDT0001: new disk, initializing [12016.770791] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12016.881943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12016.906511] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12021.369683] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [12032.562544] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12039.292907] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12048.517995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12048.998334] Lustre: lustre-OST0000: new disk, initializing [12049.008218] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12049.022548] Lustre: 351493:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12049.130378] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12054.617277] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12054.643088] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12054.818152] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12055.613854] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [12065.668369] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12106.209615] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12106.214593] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12106.226511] LustreError: Skipped 1 previous similar message [12106.229661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12106.247041] Lustre: Skipped 1 previous similar message [12109.424576] Lustre: server umount lustre-OST0000 complete [12113.890102] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12113.902420] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12116.448301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12116.458410] Lustre: Skipped 3 previous similar messages [12119.693041] Lustre: server umount lustre-MDT0000 complete [12121.572850] LustreError: 350750:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12121.587859] LustreError: 350750:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [12123.564074] LustreError: 349516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454274 with bad export cookie 7988014769211227457 [12123.565071] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12123.576625] LustreError: 349516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12124.100933] Lustre: server umount lustre-MDT0001 complete [12135.746145] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [12138.410605] Key type lgssc unregistered [12138.735875] LNet: 353667:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12138.743821] LNetError: 353667:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12138.765878] LNet: Removed LNI 192.168.206.151@tcp [12139.508160] Key type .llcrypt unregistered [12139.510587] Key type ._llcrypt unregistered [12158.872981] Key type ._llcrypt registered [12158.877699] Key type .llcrypt registered [12158.998544] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [12160.546229] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12160.578863] alg: No test for adler32 (adler32-zlib) [12161.717348] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [12162.096836] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [12163.879459] Key type lgssc registered [12165.981772] Lustre: Echo OBD driver; http://www.lustre.org/ [12175.111457] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12175.132619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12176.770532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12181.265489] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [12187.115289] LustreError: 354942:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12187.146056] LustreError: 354942:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12188.135599] LustreError: 354943:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12190.476073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12190.751634] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12195.030958] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [12202.633849] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12208.908857] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12217.130758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12217.668273] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12224.812118] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [12232.037874] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12248.545754] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12248.569569] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12248.598868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12250.592083] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12250.605089] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12250.605218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12253.641044] Lustre: server umount lustre-OST0000 complete [12260.837214] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12260.842708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12260.855258] Lustre: Skipped 1 previous similar message [12260.878217] Lustre: Skipped 2 previous similar messages [12264.595574] Lustre: server umount lustre-MDT0000 complete [12265.959055] LustreError: 354943:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12265.980416] LustreError: 354943:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12268.794870] LustreError: 354923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454419 with bad export cookie 13316560856460707462 [12268.813900] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12269.091559] Lustre: server umount lustre-MDT0001 complete [12279.616297] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [12283.516771] Key type lgssc unregistered [12283.946780] LNet: 360536:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12283.962735] LNetError: 360536:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12284.000762] LNet: Removed LNI 192.168.206.151@tcp [12285.174264] Key type .llcrypt unregistered [12285.178757] Key type ._llcrypt unregistered [12306.973841] Key type ._llcrypt registered [12306.975889] Key type .llcrypt registered [12307.113848] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [12308.350916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12308.370205] alg: No test for adler32 (adler32-zlib) [12310.024736] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [12310.432888] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [12312.210246] Key type lgssc registered [12313.715807] Lustre: Echo OBD driver; http://www.lustre.org/ [12322.306816] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12322.332377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12324.093622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12329.894902] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [12333.548980] LustreError: 363989:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12333.572435] LustreError: 363989:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12338.659891] LustreError: 363991:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12343.114346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12343.529868] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12348.843704] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [12357.435239] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12364.326882] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12371.820907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12372.140092] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12378.589017] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [12386.351492] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12409.313294] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12409.316551] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12409.365250] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12411.968619] Lustre: server umount lustre-OST0000 complete [12419.565902] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12419.569068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12419.583218] Lustre: Skipped 2 previous similar messages [12419.634402] Lustre: Skipped 2 previous similar messages [12424.189775] Lustre: server umount lustre-MDT0000 complete [12424.676574] LustreError: 363982:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12424.707702] LustreError: 363982:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12429.717277] LustreError: 363934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454580 with bad export cookie 9650158153505867044 [12429.722633] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12429.735378] LustreError: 363934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12430.576527] Lustre: server umount lustre-MDT0001 complete [12443.108947] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [12446.372859] Key type lgssc unregistered [12446.705035] LNet: 371677:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12446.714407] LNetError: 371677:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12446.734447] LNet: Removed LNI 192.168.206.151@tcp [12448.097810] Key type .llcrypt unregistered [12448.105049] Key type ._llcrypt unregistered [12470.813825] Key type ._llcrypt registered [12470.816263] Key type .llcrypt registered [12470.943891] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [12472.726601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12472.773258] alg: No test for adler32 (adler32-zlib) [12473.973440] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [12474.485655] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [12476.419129] Key type lgssc registered [12478.250776] Lustre: Echo OBD driver; http://www.lustre.org/ [12490.058979] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12490.120496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12491.743221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12497.620697] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [12501.986933] LustreError: 375342:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12502.016946] LustreError: 375342:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12507.105914] LustreError: 375344:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12509.640733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12510.192396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12515.641251] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [12525.438043] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12531.808798] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12539.640283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12540.090258] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12547.884513] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [12557.129339] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12572.143273] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12572.144903] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12572.147144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12572.156494] Lustre: Skipped 1 previous similar message [12577.250662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12577.266536] Lustre: Skipped 2 previous similar messages [12578.553713] Lustre: server umount lustre-OST0000 complete [12586.981390] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12586.988430] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12587.017435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12592.615271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12592.636097] Lustre: Skipped 3 previous similar messages [12597.215347] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12597.739466] LustreError: 375334:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12597.782386] LustreError: 375334:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12597.952667] Lustre: server umount lustre-MDT0000 complete [12602.849153] LustreError: 375342:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12602.885482] LustreError: 375342:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12603.682475] LustreError: 375280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454754 with bad export cookie 5181012923549590653 [12603.692291] LustreError: 375280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12603.693622] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12604.388553] Lustre: server umount lustre-MDT0001 complete [12618.857212] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [12622.028632] Key type lgssc unregistered [12622.375823] LNet: 381369:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12622.390737] LNetError: 381369:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12622.426901] LNet: Removed LNI 192.168.206.151@tcp [12623.318067] Key type .llcrypt unregistered [12623.321078] Key type ._llcrypt unregistered [12729.366510] Key type ._llcrypt registered [12729.378066] Key type .llcrypt registered [12729.521327] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [12730.186415] Key type .llcrypt unregistered [12730.192695] Key type ._llcrypt unregistered [12809.253875] Key type ._llcrypt registered [12809.263442] Key type .llcrypt registered [12809.402409] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [12809.852522] Key type .llcrypt unregistered [12809.859240] Key type ._llcrypt unregistered [12844.168287] Key type ._llcrypt registered [12844.173392] Key type .llcrypt registered [12844.375516] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [12844.779293] Key type .llcrypt unregistered [12844.784041] Key type ._llcrypt unregistered [12877.093178] Key type ._llcrypt registered [12877.096449] Key type .llcrypt registered [12877.229348] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [12877.607387] Key type .llcrypt unregistered [12877.610088] Key type ._llcrypt unregistered [12954.130440] Key type ._llcrypt registered [12954.132695] Key type .llcrypt registered [12954.225095] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [12954.571263] Key type .llcrypt unregistered [12954.575124] Key type ._llcrypt unregistered [12990.063357] Key type ._llcrypt registered [12990.066575] Key type .llcrypt registered [12990.168227] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [12990.515542] Key type .llcrypt unregistered [12990.520339] Key type ._llcrypt unregistered [13031.852641] Key type ._llcrypt registered [13031.866189] Key type .llcrypt registered [13032.009904] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [13032.404361] Key type .llcrypt unregistered [13032.406049] Key type ._llcrypt unregistered [13067.697253] Key type ._llcrypt registered [13067.702023] Key type .llcrypt registered [13067.858067] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [13068.303526] Key type .llcrypt unregistered [13068.307434] Key type ._llcrypt unregistered [13106.511592] Key type ._llcrypt registered [13106.513670] Key type .llcrypt registered [13106.686983] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [13107.181384] Key type .llcrypt unregistered [13107.190341] Key type ._llcrypt unregistered [13143.832276] Key type ._llcrypt registered [13143.834966] Key type .llcrypt registered [13143.953635] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [13144.349303] Key type .llcrypt unregistered [13144.355837] Key type ._llcrypt unregistered [13217.593681] Key type ._llcrypt registered [13217.596082] Key type .llcrypt registered [13217.686183] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [13217.960288] Key type .llcrypt unregistered [13217.963309] Key type ._llcrypt unregistered [13251.650859] Key type ._llcrypt registered [13251.654596] Key type .llcrypt registered [13251.759567] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [13252.185337] Key type .llcrypt unregistered [13252.188577] Key type ._llcrypt unregistered [13287.816350] Key type ._llcrypt registered [13287.820339] Key type .llcrypt registered [13288.010880] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [13288.362469] Key type .llcrypt unregistered [13288.366232] Key type ._llcrypt unregistered [13349.783746] Key type ._llcrypt registered [13349.785667] Key type .llcrypt registered [13349.879387] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [13366.676928] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [13367.493328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13367.515788] alg: No test for adler32 (adler32-zlib) [13368.626772] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [13368.880830] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [13370.583161] Key type lgssc registered [13371.647643] Lustre: Echo OBD driver; http://www.lustre.org/ [13409.992275] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13410.075888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13411.542269] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13411.587541] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13411.724498] Lustre: tst250fs-MDT0000: new disk, initializing [13411.818900] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13411.857715] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13417.114213] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [13429.230421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13429.460312] Lustre: 402009:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13429.626980] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13429.638972] Lustre: Skipped 1 previous similar message [13429.826756] Lustre: tst250fs-MDT0001: new disk, initializing [13430.016958] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13430.110486] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13430.128388] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13435.561564] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [13440.389211] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13447.785976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13448.181107] Lustre: tst250fs-OST0000: new disk, initializing [13448.186259] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13448.199882] Lustre: 403642:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13448.363285] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13454.949354] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [13455.986271] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13455.996854] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13456.155775] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13465.929468] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13466.075520] Lustre: tst250fs-OST0001: new disk, initializing [13466.079738] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13466.087308] Lustre: 404515:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13466.182574] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13472.639805] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [13473.967380] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13473.987700] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13474.099688] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13478.479072] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13480.481704] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [13480.491635] Lustre: Skipped 1 previous similar message [13499.876023] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13499.881868] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13499.900975] Lustre: Skipped 1 previous similar message [13499.914344] Lustre: Skipped 3 previous similar messages [13504.998440] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13505.010355] Lustre: Skipped 3 previous similar messages [13505.413352] Lustre: server umount tst250fs-MDT0000 complete [13508.692962] LustreError: 402000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788455659 with bad export cookie 17060125406557424719 [13508.695413] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13508.705929] LustreError: 402000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13509.086994] Lustre: server umount tst250fs-MDT0001 complete [13522.951532] Lustre: server umount tst250fs-OST0000 complete [13525.471153] Lustre: 400231:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788455660/real 1788455660] req@ffff90ec6e920a80 x1875331534360192/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788455676 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13525.509522] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13525.531182] Lustre: Skipped 2 previous similar messages [13526.761232] Lustre: server umount tst250fs-OST0001 complete [13545.601953] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [13555.984463] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [13594.470900] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 13:15:42 (1788455742) [13596.180589] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13615.794574] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [13627.371842] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [13664.065332] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13420 sec ========= 13:16:52 (1788455812) [13667.479243] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 13:16:54 (1788455814) === [13673.099570] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 13:17:01 (1788455821) === [13721.936801] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [13725.554516] Key type lgssc unregistered [13725.913542] LNet: 413662:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13725.918510] LNetError: 413662:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13725.955824] LNet: Removed LNI 192.168.206.151@tcp [13726.997150] Key type .llcrypt unregistered [13726.999120] Key type ._llcrypt unregistered