[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 1377623429 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003273] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.009000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011128] LSM: Security Framework initializing [ 0.012035] Yama: becoming mindful. [ 0.013047] SELinux: Initializing. [ 0.014051] *** VALIDATE selinux *** [ 0.024434] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.030317] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.031137] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032143] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033532] *** VALIDATE tmpfs *** [ 0.037024] *** VALIDATE proc *** [ 0.038330] *** VALIDATE cgroup *** [ 0.039018] *** VALIDATE cgroup2 *** [ 0.040248] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.041175] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.042007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.043029] Spectre V2 : User space: Vulnerable [ 0.044011] Speculative Store Bypass: Vulnerable [ 0.047251] debug: unmapping init [mem 0xffffffff91a59000-0xffffffff91a60fff] [ 0.050286] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.051768] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.052025] ... version: 2 [ 0.053013] ... bit width: 48 [ 0.054010] ... generic registers: 4 [ 0.055012] ... value mask: 0000ffffffffffff [ 0.056013] ... max period: 00007fffffffffff [ 0.057013] ... fixed-purpose events: 3 [ 0.058012] ... event mask: 000000070000000f [ 0.059314] rcu: Hierarchical SRCU implementation. [ 0.062528] smp: Bringing up secondary CPUs ... [ 0.063664] x86: Booting SMP configuration: [ 0.064027] .... node #0, CPUs: #1 #2 #3 [ 0.075638] smp: Brought up 1 node, 4 CPUs [ 0.077016] smpboot: Max logical packages: 1 [ 0.078015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.124422] node 0 deferred pages initialised in 45ms [ 0.134077] devtmpfs: initialized [ 0.135224] x86/mm: Memory block size: 128MB [ 0.141246] gcov: version magic: 0x41383552 [ 0.144358] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.146073] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.147831] pinctrl core: initialized pinctrl subsystem [ 0.148221] [ 0.149010] ************************************************************* [ 0.150038] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.151015] ** ** [ 0.152015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.153018] ** ** [ 0.154018] ** This means that this kernel is built to expose internal ** [ 0.155019] ** IOMMU data structures, which may compromise security on ** [ 0.156013] ** your system. ** [ 0.157013] ** ** [ 0.158013] ** If you see this message and you are not debugging the ** [ 0.159013] ** kernel, report this immediately to your vendor! ** [ 0.160012] ** ** [ 0.161015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.162010] ************************************************************* [ 0.163817] NET: Registered protocol family 16 [ 0.164491] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.167074] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.175923] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.181100] cpuidle: using governor menu [ 0.266299] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.268540] PCI: Using configuration type 1 for base access [ 0.271153] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.282608] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.283000] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.287057] cryptd: max_cpu_qlen set to 1000 [ 0.291325] ACPI: Added _OSI(Module Device) [ 0.292009] ACPI: Added _OSI(Processor Device) [ 0.293007] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.294007] ACPI: Added _OSI(Processor Aggregator Device) [ 0.299341] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.304554] ACPI: Interpreter enabled [ 0.306055] ACPI: PM: (supports S0 S3 S4 S5) [ 0.307009] ACPI: Using IOAPIC for interrupt routing [ 0.309114] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.311472] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.323802] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.325041] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.328014] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.331116] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.335842] acpiphp: Slot [2] registered [ 0.337108] acpiphp: Slot [5] registered [ 0.338068] acpiphp: Slot [6] registered [ 0.339100] acpiphp: Slot [7] registered [ 0.340129] acpiphp: Slot [8] registered [ 0.342126] acpiphp: Slot [9] registered [ 0.343084] acpiphp: Slot [10] registered [ 0.344195] acpiphp: Slot [3] registered [ 0.345120] acpiphp: Slot [4] registered [ 0.346061] acpiphp: Slot [11] registered [ 0.347105] acpiphp: Slot [12] registered [ 0.348063] acpiphp: Slot [13] registered [ 0.349078] acpiphp: Slot [14] registered [ 0.350142] acpiphp: Slot [15] registered [ 0.352149] acpiphp: Slot [16] registered [ 0.353095] acpiphp: Slot [17] registered [ 0.355152] acpiphp: Slot [18] registered [ 0.356130] acpiphp: Slot [19] registered [ 0.358075] acpiphp: Slot [20] registered [ 0.359083] acpiphp: Slot [21] registered [ 0.360152] acpiphp: Slot [22] registered [ 0.361067] acpiphp: Slot [23] registered [ 0.362063] acpiphp: Slot [24] registered [ 0.363046] acpiphp: Slot [25] registered [ 0.365109] acpiphp: Slot [26] registered [ 0.366105] acpiphp: Slot [27] registered [ 0.367091] acpiphp: Slot [28] registered [ 0.368097] acpiphp: Slot [29] registered [ 0.369014] acpiphp: Slot [30] registered [ 0.369893] acpiphp: Slot [31] registered [ 0.371043] PCI host bridge to bus 0000:00 [ 0.372022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.375017] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.377017] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.379019] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.382016] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.384023] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.386368] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.390439] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.395556] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.408017] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.414055] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.418022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.423022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.426020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.429669] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.433154] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.436036] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.438825] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.443010] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.455013] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.459011] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.462815] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.470014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.478017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.494019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.503286] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.511013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.517053] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.532017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.542173] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.552014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.558016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.574013] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.584173] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.591014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.597014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.611015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.621760] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.628099] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.633015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.651018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.665480] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.671014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.676013] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.692029] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.704315] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.707344] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.710377] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.712305] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.714311] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.719156] iommu: Default domain type: Passthrough [ 0.721418] SCSI subsystem initialized [ 0.722182] ACPI: bus type USB registered [ 0.724082] usbcore: registered new interface driver usbfs [ 0.725000] usbcore: registered new interface driver hub [ 0.725067] usbcore: registered new device driver usb [ 0.727052] pps_core: LinuxPPS API ver. 1 registered [ 0.729010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.730059] PTP clock support registered [ 0.732099] EDAC MC: Ver: 3.0.0 [ 0.734131] PCI: Using ACPI for IRQ routing [ 0.735861] NetLabel: Initializing [ 0.737012] NetLabel: domain hash size = 128 [ 0.740011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.742093] NetLabel: unlabeled traffic allowed by default [ 0.744132] vgaarb: loaded [ 0.746293] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.747013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.752000] clocksource: Switched to clocksource kvm-clock [ 0.861986] VFS: Disk quotas dquot_6.6.0 [ 0.863265] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.865257] *** VALIDATE ramfs *** [ 0.866250] *** VALIDATE hugetlbfs *** [ 0.867700] pnp: PnP ACPI init [ 0.870041] pnp: PnP ACPI: found 6 devices [ 0.886655] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.890688] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.893793] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.895412] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.897842] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.900893] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.905441] NET: Registered protocol family 2 [ 0.908163] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.913618] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.917417] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.923870] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.927562] TCP: Hash tables configured (established 65536 bind 65536) [ 0.930569] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.934059] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.938196] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.941617] NET: Registered protocol family 1 [ 0.944498] RPC: Registered named UNIX socket transport module. [ 0.946146] RPC: Registered udp transport module. [ 0.947647] RPC: Registered tcp transport module. [ 0.948851] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.950613] NET: Registered protocol family 44 [ 0.951893] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.953465] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.955091] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.956766] PCI: CLS 0 bytes, default 64 [ 0.958207] Unpacking initramfs... [ 2.509555] debug: unmapping init [mem 0xffff91827cc54000-0xffff91827ffbffff] [ 2.513456] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.515200] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.517346] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.043141] Initialise system trusted keyrings [ 3.044238] Key type blacklist registered [ 3.046233] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.054426] zbud: loaded [ 3.058527] *** VALIDATE nfs *** [ 3.059489] *** VALIDATE nfs4 *** [ 3.061065] pstore: using deflate compression [ 3.063985] Platform Keyring initialized [ 3.170562] NET: Registered protocol family 38 [ 3.173858] Key type asymmetric registered [ 3.175664] Asymmetric key parser 'x509' registered [ 3.177785] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.181275] io scheduler mq-deadline registered [ 3.182896] io scheduler kyber registered [ 3.184631] io scheduler bfq registered [ 3.186510] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.189935] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.193219] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.196266] ACPI: Power Button [PWRF] [ 3.202141] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.209876] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.243815] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.262724] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.314981] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.342122] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.373679] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.378646] Non-volatile memory driver v1.3 [ 3.380492] Linux agpgart interface v0.103 [ 3.514859] virtio_blk virtio1: [vda] 145880 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.518571] vda: detected capacity change from 0 to 74690560 [ 3.535750] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.538495] vdb: detected capacity change from 0 to 1073741824 [ 3.555938] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.559544] vdc: detected capacity change from 0 to 2621440000 [ 3.574183] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.577337] vdd: detected capacity change from 0 to 2621440000 [ 3.603467] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.606773] vde: detected capacity change from 0 to 4294967296 [ 3.622861] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.626541] vdf: detected capacity change from 0 to 4294967296 [ 3.634585] libphy: Fixed MDIO Bus: probed [ 3.650549] usbcore: registered new interface driver usbserial_generic [ 3.653332] usbserial: USB Serial support registered for generic [ 3.655968] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.661412] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.663408] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.666085] mousedev: PS/2 mouse device common for all mice [ 3.668638] rtc_cmos 00:05: RTC can wake from S4 [ 3.672161] rtc_cmos 00:05: registered as rtc0 [ 3.674709] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.676854] intel_pstate: CPU model not supported [ 3.678119] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.692422] hid: raw HID events driver (C) Jiri Kosina [ 3.695097] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.696384] usbcore: registered new interface driver usbhid [ 3.696388] usbhid: USB HID core driver [ 3.696493] drop_monitor: Initializing network drop monitor service [ 3.696616] Initializing XFRM netlink socket [ 3.697092] NET: Registered protocol family 10 [ 3.701235] Segment Routing with IPv6 [ 3.701279] NET: Registered protocol family 17 [ 3.701699] mpls_gso: MPLS GSO support [ 3.718774] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.724580] RAS: Correctable Errors collector initialized. [ 3.726784] AVX version of gcm_enc/dec engaged. [ 3.728509] AES CTR mode by8 optimization enabled [ 3.802512] sched_clock: Marking stable (3802402996, 0)->(5526601014, -1724198018) [ 3.807722] registered taskstats version 1 [ 3.810526] Loading compiled-in X.509 certificates [ 3.812166] zswap: loaded using pool lzo/zbud [ 4.144094] Key type big_key registered [ 4.157873] Key type encrypted registered [ 4.160848] ima: No TPM chip found, activating TPM-bypass! [ 4.163835] ima: Allocated hash algorithm: sha1 [ 4.166173] ima: No architecture policies found [ 4.168254] evm: Initialising EVM extended attributes: [ 4.170835] evm: security.selinux [ 4.172189] evm: security.ima [ 4.174265] evm: security.capability [ 4.175872] evm: HMAC attrs: 0x1 [ 4.180678] rtc_cmos 00:05: setting system clock to 2026-09-02 14:50:07 UTC (1788360607) [ 4.188280] debug: unmapping init [mem 0xffffffff92a03000-0xffffffff92bfffff] [ 4.196423] debug: unmapping init [mem 0xffffffff91782000-0xffffffff91a58fff] [ 4.207169] Write protecting the kernel read-only data: 28672k [ 4.211884] debug: unmapping init [mem 0xffffffff8fe03000-0xffffffff8fffffff] [ 4.215088] debug: unmapping init [mem 0xffffffff90714000-0xffffffff907fffff] [ 4.250798] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.258495] systemd[1]: Detected virtualization kvm. [ 4.260241] systemd[1]: Detected architecture x86-64. [ 4.261957] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.288606] systemd[1]: No hostname configured. [ 4.290170] systemd[1]: Set hostname to . [ 4.291995] random: systemd: uninitialized urandom read (16 bytes read) [ 4.294090] systemd[1]: Initializing machine ID from random generator. [ 4.348211] random: ln: uninitialized urandom read (6 bytes read) [ 4.489902] random: systemd: uninitialized urandom read (16 bytes read) [ 4.492101] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 4.495984] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.502169] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.988693] device-mapper: uevent: version 1.0.3 [ 5.991594] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System...[ 6.847937] random: fast init done [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 7.173921] virtio_net virtio0 ens2: renamed from eth0 [ 7.755326] scsi host0: ata_piix [ 7.852951] scsi host1: ata_piix [ 7.855584] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 7.863250] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 11.918061] random: crng init done [ 11.920185] random: 7 urandom warning(s) missed due to ratelimiting [ 14.451871] dracut-initqueue[586]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ 16.667369] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Reached target Remote File Systems. [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 19.476835] printk: systemd: 25 output lines suppressed due to ratelimiting [ 19.863208] SELinux: Disabled at runtime. [ 19.923671] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 19.931225] systemd[1]: Detected virtualization kvm. [ 19.932408] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 20.654417] systemd[1]: initrd-switch-root.service: Succeeded. [ 20.657897] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 20.665541] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 20.669858] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 20.673365] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 20.682230] systemd[1]: Starting Journal Service... Starting Journal Service... [ 20.686046] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice User and Session Slice. Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... [ OK ] Created slice system-getty.slice. [ 20.770524] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Mounting Kernel Debug File System... [ OK ] Stopped target Initrd File Systems. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Listening on initctl Compatibility Named Pipe. Mounting POSIX Message Queue File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 21.420218] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 22.694351] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 22.713867] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 24.475044] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 24.731309] EDAC sbridge: Ver: 1.1.2 [ 26.665087] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (6s / no limit) [** ] A start job is running for Configur…-only root support (6s / no limit)[ 27.850211] NFS: Registering the id_resolver key type [ 27.852907] Key type id_resolver registered [ 27.854809] Key type id_legacy registered [*** ] A start job is running for Configur…-only root support (7s / no limit) [ *** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Login Service. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg145-server login: [ 96.421715] libcfs: loading out-of-tree module taints kernel. [ 96.466901] Key type ._llcrypt registered [ 96.469370] Key type .llcrypt registered [ 96.561402] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 113.187877] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 114.684657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 114.701935] alg: No test for adler32 (adler32-zlib) [ 116.011297] Lustre: Lustre: Build Version: 2.17.56_2_ge76cbe5 [ 116.770237] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 118.584590] Key type lgssc registered [ 120.159101] Lustre: Echo OBD driver; http://www.lustre.org/ [ 136.250482] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 171.870610] hrtimer: interrupt took 4214370 ns [ 175.395587] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 189.229839] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 189.275810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 190.576701] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 190.614538] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 190.714790] Lustre: lustre-MDT0000: new disk, initializing [ 190.838985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 190.864838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 194.845670] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 207.196717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 207.242465] Lustre: 6518:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 207.269519] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 207.277461] Lustre: Skipped 1 previous similar message [ 207.376904] Lustre: lustre-MDT0001: new disk, initializing [ 207.436270] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 207.477816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 207.484751] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 211.428995] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 215.994633] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 224.572392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 224.806251] Lustre: lustre-OST0000: new disk, initializing [ 224.809225] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 224.814719] Lustre: 8455:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 224.871064] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 230.376557] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 231.467175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 231.478797] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 231.601337] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 244.122314] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 244.282431] Lustre: lustre-OST0001: new disk, initializing [ 244.286293] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 244.292348] Lustre: 9528:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 244.370807] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 250.299834] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 250.901375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 250.911759] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 250.993132] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 261.675454] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 268.725627] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 274.906693] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing check_logdir /tmp/testlogs/ [ 279.887911] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing yml_node [ 283.911338] Lustre: DEBUG MARKER: Client: 2.17.56.2 [ 286.080502] Lustre: DEBUG MARKER: MDS: 2.17.56.2 [ 288.260604] Lustre: DEBUG MARKER: OSS: 2.17.56.2 [ 289.858469] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Sep 2 10:54:51 EDT 2026 [ 305.842192] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 307.224496] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 322.532512] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 322.550378] Lustre: Skipped 1 previous similar message [ 322.553022] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 322.577333] Lustre: Skipped 3 previous similar messages [ 327.648740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 327.660129] Lustre: Skipped 3 previous similar messages [ 328.772606] Lustre: server umount lustre-MDT0000 complete [ 332.506390] LustreError: 6511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788360935 with bad export cookie 2165348438838709717 [ 332.514432] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 332.528774] LustreError: 6511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 332.768855] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 332.781651] Lustre: Skipped 1 previous similar message [ 332.789532] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 332.804468] Lustre: Skipped 1 previous similar message [ 337.888768] LustreError: 6525:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 337.891923] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 337.918522] LustreError: 6525:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 337.938800] Lustre: Skipped 1 previous similar message [ 339.210564] Lustre: server umount lustre-MDT0001 complete [ 343.322242] Lustre: server umount lustre-OST0000 complete [ 346.884862] Lustre: server umount lustre-OST0001 complete [ 353.010485] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 361.049530] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 392.066790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 392.273368] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 392.304945] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 392.378811] Lustre: lustre-MDT0000: new disk, initializing [ 392.461409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 392.478409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 396.646671] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 405.377159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 405.457989] Lustre: 15077:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 405.510318] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 405.513303] Lustre: Skipped 1 previous similar message [ 405.576717] Lustre: lustre-MDT0001: new disk, initializing [ 405.689189] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 405.723604] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 405.739661] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 409.592610] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 417.517944] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 422.739328] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 428.381464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 428.589785] Lustre: lustre-OST0000: new disk, initializing [ 428.592838] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 428.597948] Lustre: 17048:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 428.685552] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 430.082395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 430.098513] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 430.210081] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 434.410419] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 442.057426] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 444.925824] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 445.094539] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 448.725211] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 448.921388] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 450.528587] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 450.539721] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 450.546521] Lustre: Skipped 1 previous similar message [ 450.550052] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 456.763726] Lustre: server umount lustre-OST0000 complete [ 460.769985] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 460.771340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 460.779788] Lustre: Skipped 2 previous similar messages [ 460.782963] Lustre: Skipped 3 previous similar messages [ 465.890306] LustreError: 15956:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 465.911455] LustreError: 15956:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 465.941861] Lustre: server umount lustre-MDT0000 complete [ 468.778711] LustreError: 15069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788361072 with bad export cookie 2165348438838711250 [ 468.787949] LustreError: 15069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 468.791812] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 469.087836] Lustre: server umount lustre-MDT0001 complete [ 473.896681] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 10:57:55 (1788361075) [ 487.451988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 487.798792] LustreError: 19920:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 487.845412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 490.972292] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 493.025505] LustreError: 19921:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 497.394952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 501.154438] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 507.273528] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 511.696361] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 513.250353] Lustre: Failing over lustre-MDT0000 [ 513.523197] Lustre: server umount lustre-MDT0000 complete [ 518.114057] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 518.115255] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 518.119875] LustreError: Skipped 1 previous similar message [ 518.121507] LustreError: 19920:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 518.127345] Lustre: Skipped 1 previous similar message [ 527.329770] LustreError: 19921:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 527.345397] LustreError: 19921:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 530.200919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 530.276564] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 533.271600] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 535.523186] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 535.532023] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 535.547439] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 540.822272] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 545.761753] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 545.766393] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 545.780858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 545.785288] Lustre: Skipped 1 previous similar message [ 550.143977] Lustre: server umount lustre-MDT0000 complete [ 551.908688] LustreError: 21938:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 551.917879] LustreError: 21938:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 553.085333] LustreError: 19901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788361156 with bad export cookie 2165348438838712664 [ 553.091443] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 553.325406] Lustre: server umount lustre-MDT0001 complete [ 559.592146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 560.098017] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 560.100861] Lustre: Skipped 2 previous similar messages [ 563.810973] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 570.999047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 575.365281] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 581.786668] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 587.411584] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 589.041096] Lustre: Failing over lustre-MDT0000 [ 589.224424] Lustre: server umount lustre-MDT0000 complete [ 590.819908] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 590.835759] LustreError: 23248:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 590.839967] Lustre: Skipped 2 previous similar messages [ 590.871542] LustreError: 23248:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 606.438638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 606.541860] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 606.755844] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 610.100753] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 611.812907] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 611.814122] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 611.816054] Lustre: Skipped 1 previous similar message [ 611.840796] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 617.170496] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 622.049515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 622.052674] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 622.055421] Lustre: Skipped 3 previous similar messages [ 634.335227] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 634.502443] Lustre: server umount lustre-MDT0000 complete [ 637.493759] LustreError: 23235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788361240 with bad export cookie 2165348438838713602 [ 637.494327] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 637.503861] LustreError: 23235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 637.742262] Lustre: server umount lustre-MDT0001 complete [ 643.970722] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 11:00:45 (1788361245) [ 656.504777] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 663.556862] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 692.115182] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 700.793504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 700.936870] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 700.994381] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 701.052792] Lustre: lustre-MDT0000: new disk, initializing [ 701.112696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 701.117549] Lustre: Skipped 2 previous similar messages [ 701.128873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 704.558880] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 713.526959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 713.594527] Lustre: 29635:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 713.843173] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 717.645962] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 724.509720] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 728.793727] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 733.940080] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 734.144169] Lustre: 31602:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 735.350293] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 735.390941] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 735.434275] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 735.442881] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 739.252377] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 748.030137] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 752.624721] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 771.557664] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 771.557829] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 771.563776] LustreError: Skipped 1 previous similar message [ 771.565133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 771.573563] Lustre: Skipped 3 previous similar messages [ 771.594430] Lustre: Skipped 5 previous similar messages [ 774.802519] Lustre: server umount lustre-OST0000 complete [ 780.257726] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 784.055808] Lustre: server umount lustre-MDT0000 complete [ 786.803918] LustreError: 29629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788361390 with bad export cookie 2165348438838714008 [ 786.813400] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 786.814122] LustreError: 29629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 787.189565] Lustre: server umount lustre-MDT0001 complete [ 795.725934] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 797.805943] Key type lgssc unregistered [ 798.032554] LNet: 33576:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 798.036423] LNetError: 33576:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 798.050582] LNet: Removed LNI 192.168.201.145@tcp [ 798.643161] Key type .llcrypt unregistered [ 798.644979] Key type ._llcrypt unregistered [ 816.613673] Key type ._llcrypt registered [ 816.615655] Key type .llcrypt registered [ 816.702270] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 817.531119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 817.620701] alg: No test for adler32 (adler32-zlib) [ 818.663250] Lustre: Lustre: Build Version: 2.17.56_2_ge76cbe5 [ 818.849342] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 820.495142] Key type lgssc registered [ 821.466692] Lustre: Echo OBD driver; http://www.lustre.org/ [ 827.022856] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 827.034415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 828.380127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 831.796278] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 838.063167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 838.295709] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 841.158937] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 846.225262] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 850.260049] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 855.829337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 856.227574] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 860.993506] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 863.412486] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 867.019763] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 874.686468] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 888.056959] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 11:04:49 (1788361489) [ 889.673952] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 891.315639] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 896.538830] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 11:04:58 (1788361498) [ 902.571822] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 11:05:04 (1788361504) [ 903.267748] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 909.503608] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 909.510108] Lustre: Skipped 2 previous similar messages [ 911.094404] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 11:05:12 (1788361512) [ 914.056586] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 914.059480] Lustre: Skipped 3 previous similar messages [ 926.763908] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 926.769919] Lustre: Skipped 6 previous similar messages [ 928.428845] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 11:05:30 (1788361530) [ 930.320566] Lustre: *** cfs_fail_loc=131b, val=2*** [ 932.448888] Lustre: *** cfs_fail_loc=131b, val=2*** [ 938.276744] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 11:05:40 (1788361540) [ 942.827445] Lustre: Disabling parameter general.jobid_name=TESTNAME1 in log params [ 942.834685] Lustre: Skipped 6 previous similar messages [ 948.875447] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 11:05:50 (1788361550) [ 956.895988] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 956.897396] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 956.912157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 956.917944] Lustre: Skipped 1 previous similar message [ 962.025763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 962.028121] Lustre: Skipped 1 previous similar message [ 962.837224] Lustre: server umount lustre-OST0000 complete [ 966.624186] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 966.627397] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 966.634154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 966.637511] Lustre: Skipped 1 previous similar message [ 971.477117] Lustre: server umount lustre-MDT0000 complete [ 972.256910] LustreError: 34963:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 972.273473] LustreError: 34963:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 973.948931] LustreError: 34950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788361577 with bad export cookie 9454132203899580462 [ 973.956089] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 973.959946] LustreError: 34950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 974.141595] Lustre: server umount lustre-MDT0001 complete [ 980.948275] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 982.822409] Key type lgssc unregistered [ 983.010589] LNet: 42825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 983.020943] LNetError: 42825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 983.038767] LNet: Removed LNI 192.168.201.145@tcp [ 983.542808] Key type .llcrypt unregistered [ 983.544849] Key type ._llcrypt unregistered [ 995.492044] Key type ._llcrypt registered [ 995.495333] Key type .llcrypt registered [ 995.570943] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 1006.545225] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 1007.263487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1007.332031] alg: No test for adler32 (adler32-zlib) [ 1008.420210] Lustre: Lustre: Build Version: 2.17.56_2_ge76cbe5 [ 1008.612861] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 1010.267173] Key type lgssc registered [ 1010.928867] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1034.426699] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1034.456452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1035.628879] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1035.655077] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1035.718172] Lustre: lustre-MDT0000: new disk, initializing [ 1035.781024] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1035.794710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1038.880788] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1047.639646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1047.755470] Lustre: 45972:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1047.802695] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1047.806897] Lustre: Skipped 1 previous similar message [ 1047.887571] Lustre: lustre-MDT0001: new disk, initializing [ 1047.971932] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1048.012758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1048.027708] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1051.357196] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1058.213981] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1061.753980] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1066.545709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1066.770901] Lustre: lustre-OST0000: new disk, initializing [ 1066.775708] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1066.781810] Lustre: 47941:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1066.848686] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1071.458656] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1074.785271] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1074.790147] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1074.877729] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1079.251556] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1082.745899] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1082.901730] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1086.393354] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1086.559838] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1090.530101] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1090.543915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1090.548059] Lustre: Skipped 1 previous similar message [ 1090.560968] Lustre: Skipped 1 previous similar message [ 1095.656824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1095.664708] Lustre: Skipped 1 previous similar message [ 1100.774191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1100.782120] Lustre: Skipped 1 previous similar message [ 1102.303116] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1102.413904] Lustre: server umount lustre-OST0000 complete [ 1105.892948] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1105.896918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1105.898903] Lustre: Skipped 1 previous similar message [ 1105.908294] Lustre: Skipped 1 previous similar message [ 1111.009154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1111.015195] Lustre: Skipped 1 previous similar message [ 1111.248650] Lustre: server umount lustre-MDT0000 complete [ 1113.440067] LustreError: 45963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788361716 with bad export cookie 15428109702243903489 [ 1113.455458] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1113.669235] Lustre: server umount lustre-MDT0001 complete [ 1119.213257] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 11:08:41 (1788361721) [ 1123.585414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1123.988306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1126.713944] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1129.448860] LustreError: 50399:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1129.471109] LustreError: 50399:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1132.538297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1132.784327] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1135.558801] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1139.563519] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1142.977260] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1147.583780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1147.789355] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1151.780461] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1156.971776] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1160.938460] LustreError: 52926:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1161.617749] Lustre: Modifying parameter general.timeout=1 in log params [ 1162.880708] Lustre: Modifying parameter general.timeout=3 in log params [ 1162.883692] Lustre: Skipped 1 previous similar message [ 1165.364600] Lustre: Modifying parameter general.timeout=7 in log params [ 1165.368983] Lustre: Skipped 3 previous similar messages [ 1169.927748] Lustre: Modifying parameter general.timeout=14 in log params [ 1169.931285] Lustre: Skipped 6 previous similar messages [ 1178.033129] Lustre: Modifying parameter general.timeout=28 in log params [ 1178.035739] Lustre: Skipped 13 previous similar messages [ 1194.506873] Lustre: Modifying parameter general.timeout=54 in log params [ 1194.510219] Lustre: Skipped 25 previous similar messages [ 1226.816365] Lustre: Modifying parameter general.timeout=102 in log params [ 1226.818940] Lustre: Skipped 47 previous similar messages [ 1253.806591] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 11:10:55 (1788361855) [ 1255.842582] Lustre: 59847:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1257.247934] Lustre: 59958:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1260.080253] Lustre: 60155:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1260.087394] Lustre: 60155:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1280.149486] Lustre: 61714:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1280.157823] Lustre: 61714:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1280.167966] LustreError: 61714:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1282.754361] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 11:11:24 (1788361884) [ 1290.283606] Lustre: server umount lustre-MDT0000 complete [ 1291.237299] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1291.242865] LustreError: 50395:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1291.249390] Lustre: Skipped 1 previous similar message [ 1291.271500] LustreError: 50395:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1292.842404] LustreError: 50379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788361896 with bad export cookie 15428109702243904371 [ 1292.844437] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1292.848055] LustreError: 50379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 1295.104389] Lustre: server umount lustre-MDT0001 complete [ 1308.136829] Lustre: server umount lustre-OST0000 complete [ 1325.375933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1325.567859] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1325.605901] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1325.615143] Lustre: Skipped 50 previous similar messages [ 1325.775281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1328.941572] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1333.711965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1333.758858] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1333.775635] Lustre: 64247:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1336.637385] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1341.111901] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1344.346921] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1348.407899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1348.511843] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1348.554379] LustreError: 66089:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1352.577243] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1357.026298] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1357.960665] Lustre: 66809:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1362.916362] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1362.916389] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1362.916396] Lustre: Skipped 1 previous similar message [ 1362.923672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1362.948440] Lustre: Skipped 1 previous similar message [ 1367.549468] Lustre: server umount lustre-OST0000 complete [ 1370.084110] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1370.087271] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1376.049466] Lustre: server umount lustre-MDT0000 complete [ 1378.273841] LustreError: 64255:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1378.283340] LustreError: 64255:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1378.372967] LustreError: 64238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788361981 with bad export cookie 15428109702243907136 [ 1378.376948] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1378.387739] LustreError: 64238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1378.564093] Lustre: server umount lustre-MDT0001 complete [ 1386.356756] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 1388.248947] Key type lgssc unregistered [ 1388.447678] LNet: 68015:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1388.451640] LNetError: 68015:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1388.461231] LNet: Removed LNI 192.168.201.145@tcp [ 1388.994556] Key type .llcrypt unregistered [ 1388.996429] Key type ._llcrypt unregistered [ 1405.607714] Key type ._llcrypt registered [ 1405.609217] Key type .llcrypt registered [ 1405.703459] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 1406.520319] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1406.563975] alg: No test for adler32 (adler32-zlib) [ 1407.529783] Lustre: Lustre: Build Version: 2.17.56_2_ge76cbe5 [ 1407.706209] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 1409.343286] Key type lgssc registered [ 1410.102540] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1415.348904] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1415.365683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1416.678863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1419.504555] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1425.057943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1425.452779] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1428.433143] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1433.279398] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1437.494650] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1442.819654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1443.276832] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1447.655339] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1453.785611] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1470.188791] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1481.184972] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1481.188546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1481.193358] Lustre: Skipped 2 previous similar messages [ 1481.197182] Lustre: Skipped 1 previous similar message [ 1485.954646] Lustre: server umount lustre-MDT0000 complete [ 1486.311103] LustreError: 71256:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1486.328612] LustreError: 71256:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 1488.736958] LustreError: 69386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788362092 with bad export cookie 12803017481831439700 [ 1488.740936] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1488.744826] LustreError: 69386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1488.947680] Lustre: server umount lustre-MDT0001 complete [ 1502.064291] Lustre: server umount lustre-OST0000 complete [ 1521.898552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1522.073369] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1522.096886] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1522.277645] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1524.916372] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1531.035422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1531.123235] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1531.143575] Lustre: 74293:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1531.519305] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1534.331722] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1539.325985] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1543.562332] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1548.027099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1548.149694] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1548.391828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1552.168101] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1556.456795] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1557.587337] Lustre: 76868:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1558.746806] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1564.129938] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1564.138120] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1564.143237] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1564.149027] Lustre: Skipped 1 previous similar message [ 1567.674434] Lustre: server umount lustre-OST0000 complete [ 1572.321329] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1572.325222] LustreError: Skipped 1 previous similar message [ 1572.327711] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1572.334343] Lustre: Skipped 1 previous similar message [ 1572.337306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1574.369323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1574.370127] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1576.238948] Lustre: server umount lustre-MDT0000 complete [ 1578.415124] LustreError: 74287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788362181 with bad export cookie 12803017481831440526 [ 1578.418576] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1578.422415] LustreError: 74287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1578.636763] Lustre: server umount lustre-MDT0001 complete [ 1585.016907] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 1586.829598] Key type lgssc unregistered [ 1587.047742] LNet: 78075:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1587.052347] LNetError: 78075:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1587.070236] LNet: Removed LNI 192.168.201.145@tcp [ 1587.614195] Key type .llcrypt unregistered [ 1587.615944] Key type ._llcrypt unregistered [ 1603.335878] Key type ._llcrypt registered [ 1603.337541] Key type .llcrypt registered [ 1603.403240] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 1603.994842] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1604.004203] alg: No test for adler32 (adler32-zlib) [ 1604.935850] Lustre: Lustre: Build Version: 2.17.56_2_ge76cbe5 [ 1605.130562] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 1606.768613] Key type lgssc registered [ 1607.515690] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1612.394074] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1612.427467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1613.811181] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1616.299414] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1621.494639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1621.909465] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1624.730776] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1629.315450] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1632.802911] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1636.799023] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1637.117373] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1640.926974] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1645.180715] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1650.771083] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1651.483921] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1652.528662] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1652.531814] Lustre: Skipped 1 previous similar message [ 1655.224124] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1655.227438] Lustre: Skipped 3 previous similar messages [ 1659.612848] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1659.616782] Lustre: Skipped 6 previous similar messages [ 1667.831225] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1667.837913] Lustre: Skipped 12 previous similar messages [ 1684.150621] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1684.154152] Lustre: Skipped 27 previous similar messages [ 1718.048104] Lustre: 87565:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1718.687315] Lustre: Disabling parameter general.jobid_var= in log params [ 1718.694325] Lustre: Skipped 44 previous similar messages [ 1719.939329] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 11:18:41 (1788362321) [ 1720.947239] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1722.130119] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 11:18:44 (1788362324) [ 1726.449541] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1726.457665] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1726.474642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1729.504436] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1729.507588] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1729.511631] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1729.526430] Lustre: Skipped 1 previous similar message [ 1730.840365] Lustre: server umount lustre-OST0000 complete [ 1734.625244] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1734.640190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1734.644668] Lustre: Skipped 1 previous similar message [ 1739.746407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1739.755091] Lustre: Skipped 1 previous similar message [ 1741.904737] Lustre: server umount lustre-MDT0000 complete [ 1744.368566] LustreError: 79461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788362347 with bad export cookie 5920238329996091864 [ 1744.370710] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1744.374889] LustreError: 79461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1744.604661] Lustre: server umount lustre-MDT0001 complete [ 1751.027679] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 1752.825475] Key type lgssc unregistered [ 1753.012835] LNet: 88813:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1753.024258] LNetError: 88813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1753.034876] LNet: Removed LNI 192.168.201.145@tcp [ 1753.603716] Key type .llcrypt unregistered [ 1753.605884] Key type ._llcrypt unregistered [ 1756.242729] Key type ._llcrypt registered [ 1756.244808] Key type .llcrypt registered [ 1756.296302] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1758.932070] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules [ 1759.510776] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1759.524964] alg: No test for adler32 (adler32-zlib) [ 1760.479968] Lustre: Lustre: Build Version: 2.17.56_2_ge76cbe5 [ 1760.642650] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 1762.295183] Key type lgssc registered [ 1762.919455] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1763.660133] LustreError: 89990:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 1773.556433] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 1778.002930] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1784.375996] LustreError: 89990:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1784.404984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1785.845310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1788.581463] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1793.818361] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 11:19:55 (1788362395) [ 1796.071354] LustreError: 91006:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1796.079016] LustreError: 91006:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1796.857229] LustreError: 91038:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1796.862550] LustreError: 91038:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1796.869426] LustreError: 91038:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 11, retries 0, failed: rc = -5 [ 1797.077328] Lustre: server umount lustre-MDT0000 complete [ 1806.276917] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 1808.367660] Key type lgssc unregistered [ 1808.579449] LNet: 92529:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1808.584444] LNetError: 92529:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1809.638514] LNet: Removed LNI 192.168.201.145@tcp [ 1810.326187] Key type .llcrypt unregistered [ 1810.332650] Key type ._llcrypt unregistered [ 1823.229877] Key type ._llcrypt registered [ 1823.231747] Key type .llcrypt registered [ 1823.307944] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 1824.023726] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1824.036839] alg: No test for adler32 (adler32-zlib) [ 1824.931912] Lustre: Lustre: Build Version: 2.17.56_2_ge76cbe5 [ 1825.075811] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 1826.689466] Key type lgssc registered [ 1827.538507] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1832.623583] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1832.646964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1833.983061] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1836.828786] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1842.327184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1842.804524] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1844.965134] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1849.092383] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1852.239235] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1856.568185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1856.951895] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1860.658402] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1865.272390] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1901.696959] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 11:21:43 (1788362503) [ 1903.102870] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1904.328859] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 11:21:46 (1788362506) [ 1908.198498] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1908.210333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1908.218533] Lustre: Skipped 2 previous similar messages [ 1912.642441] Lustre: server umount lustre-MDT0000 complete [ 1913.313430] LustreError: 93726:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1913.326144] LustreError: 93726:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 1914.731314] LustreError: 93707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788362518 with bad export cookie 11769155635613872631 [ 1914.736472] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1914.743320] LustreError: 93707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1914.918672] Lustre: server umount lustre-MDT0001 complete [ 1927.345311] Lustre: server umount lustre-OST0000 complete [ 1934.056438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1934.293321] LustreError: 97995:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1934.380695] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1936.405163] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1939.425032] LustreError: 97996:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1940.962715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1941.296634] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1943.197430] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1946.977627] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1950.251349] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1957.527226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1957.638064] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1957.641940] LustreError: 97984:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1957.653864] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1957.657585] LustreError: 100077:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 1957.660445] LustreError: 100077:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 1957.662805] LustreError: 100077:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1957.695677] Lustre: server umount lustre-OST0000 complete [ 1957.697213] LustreError: 100077:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1962.196699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1962.303643] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1962.308200] LustreError: 97984:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1962.315726] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1962.325360] LustreError: 100541:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 1962.332273] LustreError: 100541:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 1962.337978] LustreError: 100541:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1962.379860] Lustre: server umount lustre-OST0000 complete [ 1962.383347] LustreError: 100541:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1966.813207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1966.914937] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1966.917766] Lustre: Found index 0 for lustre-OST0000, updating log [ 1966.920627] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1966.952404] Lustre: lustre-OST0000: new disk, initializing [ 1966.955081] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1966.958599] Lustre: 101069:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1967.090539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1970.969303] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 1973.509987] Lustre: Failing over lustre-OST0000 [ 1973.632124] Lustre: server umount lustre-OST0000 complete [ 1976.125207] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1976.130708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1976.133037] Lustre: Skipped 3 previous similar messages [ 1976.140160] Lustre: Skipped 1 previous similar message [ 1981.409027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1982.438810] Lustre: server umount lustre-MDT0000 complete [ 1984.922454] LustreError: 97977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788362588 with bad export cookie 11769155635613875382 [ 1984.924271] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1984.934346] LustreError: 97977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1985.130658] Lustre: server umount lustre-MDT0001 complete [ 1990.157283] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 11:23:12 (1788362592) [ 1998.253860] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 2003.618540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2003.832179] LustreError: 103411:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2003.893836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2005.933519] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2009.055880] LustreError: 103412:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2010.492559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2013.097259] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2014.862865] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2018.431407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2021.572578] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2025.695777] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2025.734891] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2025.782357] Lustre: lustre-OST0001: new disk, initializing [ 2025.784375] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2025.786762] Lustre: 105725:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2027.612502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2027.616794] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2027.664602] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2028.547397] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2032.690093] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2035.919382] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2039.268790] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2039.275873] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2039.276862] Lustre: Skipped 1 previous similar message [ 2039.285813] Lustre: Skipped 3 previous similar messages [ 2043.690703] Lustre: server umount lustre-MDT0001 complete [ 2044.385644] LustreError: 103407:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2044.392120] LustreError: 103407:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2047.659254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2047.728804] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2047.731186] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2047.753686] Lustre: 103400:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2048.080546] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2048.084067] Lustre: Skipped 3 previous similar messages [ 2049.708922] LustreError: 107445:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2049.712854] LustreError: 107445:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2049.715760] LustreError: 103403:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2049.773755] LustreError: 107463:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.145@tcp: cfg command failed: rc = -17 [ 2049.777941] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2049.777941] [ 2049.880417] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2053.098917] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2053.759443] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 11:24:15 (1788362655) [ 2061.031438] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 2068.048491] Lustre: 109328:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2077.866443] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2079.349842] Lustre: 110461:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2155.489324] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2155.490018] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2155.491590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2155.491597] Lustre: Skipped 1 previous similar message [ 2155.518567] Lustre: Skipped 3 previous similar messages [ 2156.827384] Lustre: server umount lustre-MDT0000 complete [ 2159.099808] LustreError: 104550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788362762 with bad export cookie 11769155635613876222 [ 2159.103112] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2159.104759] LustreError: 104550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2159.291085] Lustre: server umount lustre-MDT0001 complete [ 2170.872987] Lustre: server umount lustre-OST0000 complete [ 2183.394920] Lustre: server umount lustre-OST0001 complete [ 2187.151921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2192.316892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2197.699882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2203.741205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2215.677399] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 2222.318977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2222.336892] Lustre: lustre-MDT0000: reset Object Index mappings [ 2222.599605] LustreError: 114406:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2222.613992] LustreError: 114406:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2222.694136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2225.893448] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2231.679257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2231.701351] Lustre: lustre-MDT0001: reset Object Index mappings [ 2234.673242] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2236.603957] Lustre: 115554:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2240.369440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2241.905469] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 2244.351404] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2248.175357] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 2249.848793] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2254.523775] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2255.335830] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 2255.341587] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 2260.448388] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2262.814288] Lustre: 117427:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2294.024903] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 11:28:15 (1788362895) [ 2321.891702] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2321.899496] Lustre: Skipped 1 previous similar message [ 2321.913067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2321.917843] Lustre: Skipped 6 previous similar messages [ 2336.223126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2336.386484] Lustre: server umount lustre-MDT0000 complete [ 2337.255066] LustreError: 114407:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2337.266628] LustreError: 114407:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 2338.988609] LustreError: 114387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788362942 with bad export cookie 11769155635613923059 [ 2338.992915] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2338.996401] LustreError: 114387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2342.370922] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2342.377948] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2342.384032] Lustre: Skipped 4 previous similar messages [ 2342.390483] Lustre: Skipped 8 previous similar messages [ 2345.403636] Lustre: server umount lustre-MDT0001 complete [ 2362.335115] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2362.388855] Lustre: server umount lustre-OST0000 complete [ 2370.910770] Lustre: server umount lustre-OST0001 complete [ 2374.909940] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 2380.256379] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 2403.498389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2403.659495] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2403.676925] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2403.748796] Lustre: lustre-MDT0000: new disk, initializing [ 2403.804522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2403.810765] Lustre: Skipped 3 previous similar messages [ 2403.817374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2406.433181] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2411.338971] Lustre: server umount lustre-MDT0000 complete [ 2417.047823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2417.192430] Lustre: Found index 0 for lustre-MDT0000, updating log [ 2417.198323] Lustre: 121701:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2417.204819] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 2419.363665] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2421.815707] Lustre: server umount lustre-MDT0000 complete [ 2426.756130] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 11:30:28 (1788363028) [ 2427.703701] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 2428.758548] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 11:30:30 (1788363030) [ 2442.083698] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 2449.972697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2453.329816] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2458.747361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2458.837204] Lustre: 123749:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2458.876921] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2458.880972] Lustre: Skipped 1 previous similar message [ 2458.929486] Lustre: lustre-MDT0001: new disk, initializing [ 2459.009725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2459.019603] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2462.101333] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2465.803162] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2470.131883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2470.305506] Lustre: lustre-OST0000: new disk, initializing [ 2470.308045] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2470.312078] Lustre: 125322:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2472.055750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2472.069189] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2472.150951] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2474.309602] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2481.696301] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2481.856488] Lustre: lustre-OST0001: new disk, initializing [ 2481.859180] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2481.863995] Lustre: 126193:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2483.581071] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2483.593628] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2483.669307] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2486.408667] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2493.414708] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2495.398406] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2743.314635] Lustre: 132591:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 2743.319363] Lustre: 132591:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 2743.322972] Lustre: 132591:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 2743.327154] Lustre: 132591:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 2743.332425] Lustre: 132591:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 2743.337102] Lustre: 132591:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 2915.509030] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 11:38:37 (1788363517) [ 2930.659651] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2930.664448] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2930.675118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2930.677830] Lustre: Skipped 2 previous similar messages [ 2933.794381] Lustre: server umount lustre-MDT0000 complete [ 2935.540222] LustreError: 132604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788363538 with bad export cookie 11769155635613984722 [ 2935.545013] LustreError: 132604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2935.545460] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2937.313286] LustreError: 126989:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2937.321265] LustreError: 126989:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2941.835699] Lustre: server umount lustre-MDT0001 complete [ 2958.303183] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2958.405991] Lustre: server umount lustre-OST0000 complete [ 2966.690889] Lustre: server umount lustre-OST0001 complete [ 2969.544890] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 2973.988339] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 2991.128694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2991.256265] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2991.278053] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2991.318451] Lustre: lustre-MDT0000: new disk, initializing [ 2991.352940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2991.355616] Lustre: Skipped 5 previous similar messages [ 2991.364770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2993.277976] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 2998.961559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2999.016824] Lustre: 135616:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2999.196505] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3001.332606] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3005.935345] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3008.587789] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3011.943728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3012.090748] Lustre: 137585:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3013.851349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3013.861371] Lustre: Skipped 1 previous similar message [ 3013.865729] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3013.924467] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3015.258741] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3020.191935] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3024.392960] Lustre: lustre-MDD0000: changelog on [ 3025.631125] Lustre: lustre-MDD0001: changelog on [ 3721.770371] Lustre: lustre-MDD0001: changelog off [ 3722.477031] Lustre: 135624:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 3722.831334] Lustre: lustre-MDD0000: changelog off [ 3725.793219] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3725.797780] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3725.798376] LustreError: Skipped 1 previous similar message [ 3725.801191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3725.805498] Lustre: Skipped 6 previous similar messages [ 3725.814819] Lustre: Skipped 9 previous similar messages [ 3736.037046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3736.040863] Lustre: Skipped 3 previous similar messages [ 3739.103188] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3739.163771] Lustre: server umount lustre-OST0000 complete [ 3741.152624] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3741.155949] Lustre: Skipped 1 previous similar message [ 3746.757761] Lustre: server umount lustre-MDT0000 complete [ 3748.121310] LustreError: 135611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364351 with bad export cookie 11769155635614800390 [ 3748.123221] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3748.125513] LustreError: 135611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3748.251296] Lustre: server umount lustre-MDT0001 complete [ 3752.229550] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 3753.394117] Key type lgssc unregistered [ 3753.544395] LNet: 146439:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3753.546667] LNetError: 146439:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3753.557365] LNet: Removed LNI 192.168.201.145@tcp [ 3753.921160] Key type .llcrypt unregistered [ 3753.922896] Key type ._llcrypt unregistered [ 3762.067736] Key type ._llcrypt registered [ 3762.068929] Key type .llcrypt registered [ 3762.113136] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 3769.033276] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 3769.373696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3769.380522] alg: No test for adler32 (adler32-zlib) [ 3770.262551] Lustre: Lustre: Build Version: 2.17.56_2_ge76cbe5 [ 3770.369742] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 3771.960402] Key type lgssc registered [ 3772.474059] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3785.889405] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3785.897744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3786.995995] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3787.012273] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3787.052599] Lustre: lustre-MDT0000: new disk, initializing [ 3787.096933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3787.108039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3788.775841] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3793.342887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3793.392264] Lustre: 149614:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3793.416236] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3793.418431] Lustre: Skipped 1 previous similar message [ 3793.460076] Lustre: lustre-MDT0001: new disk, initializing [ 3793.496757] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3793.510385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3793.516694] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3794.905210] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3799.099718] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3801.555471] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3804.661916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3804.819715] Lustre: lustre-OST0000: new disk, initializing [ 3804.822358] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3804.832850] Lustre: 151582:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3804.896545] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3805.142690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3805.152227] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3805.228146] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3807.860447] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3814.147151] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3817.797624] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_module obdecho/obdecho [ 3819.076508] LustreError: 152585:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 3819.080667] LustreError: 152585:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 3820.298134] LustreError: 152682:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 3829.217683] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3829.224415] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3829.234808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3830.752802] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3830.755588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3830.769498] Lustre: Skipped 1 previous similar message [ 3830.784946] Lustre: Skipped 2 previous similar messages [ 3833.360460] Lustre: server umount lustre-MDT0000 complete [ 3835.416419] LustreError: 149607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364438 with bad export cookie 9218570464430317094 [ 3835.420962] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3835.422837] LustreError: 149607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3835.648771] Lustre: server umount lustre-MDT0001 complete [ 3848.570751] Lustre: server umount lustre-OST0000 complete [ 3859.236122] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 3863.461910] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 3883.336513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3883.464162] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3883.483965] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3883.524364] Lustre: lustre-MDT0000: new disk, initializing [ 3883.576903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3883.591223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3886.182749] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3892.864308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3892.949713] Lustre: 156390:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3892.983607] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3892.987588] Lustre: Skipped 1 previous similar message [ 3893.051123] Lustre: lustre-MDT0001: new disk, initializing [ 3893.099744] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3893.117923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3893.124908] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3895.534677] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3901.537279] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3905.082726] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3908.926542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3909.198208] Lustre: lustre-OST0000: new disk, initializing [ 3909.201445] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3909.206433] Lustre: 158357:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3909.276806] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3912.894165] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3915.801297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3915.810908] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3915.876147] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3918.584933] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3921.267949] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3921.388926] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3924.059681] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3924.215406] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3925.985830] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3925.985958] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3925.995962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3925.996863] LustreError: Skipped 1 previous similar message [ 3931.106309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3931.109336] Lustre: Skipped 2 previous similar messages [ 3931.230434] Lustre: server umount lustre-OST0000 complete [ 3934.175859] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3934.178216] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3934.183598] Lustre: Skipped 1 previous similar message [ 3936.225355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3936.228112] Lustre: Skipped 2 previous similar messages [ 3938.853472] Lustre: server umount lustre-MDT0000 complete [ 3940.404484] LustreError: 156381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364543 with bad export cookie 9218570464430318039 [ 3940.406737] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3940.409187] LustreError: 156381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3940.549871] Lustre: server umount lustre-MDT0001 complete [ 3942.019888] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 11:55:44 (1788364544) [ 3942.681856] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 3943.399322] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 11:55:45 (1788364545) [ 3946.714167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3947.019676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3948.747277] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3952.096707] LustreError: 160766:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3952.102845] LustreError: 160766:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3952.520761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3954.471085] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3957.755717] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3960.390623] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3963.766923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3966.714549] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 3970.338809] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3986.278131] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.45@tcp (stopping) [ 3987.040524] Lustre: server umount lustre-MDT0000 complete [ 3987.424787] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3987.426920] LustreError: 160766:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3987.430210] Lustre: Skipped 3 previous similar messages [ 3987.435566] LustreError: 160766:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3988.524681] LustreError: 162606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364591 with bad export cookie 9218570464430318921 [ 3988.525701] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3988.528975] LustreError: 162606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3991.390532] LustreError: 160763:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3991.407135] LustreError: 160763:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3993.568098] LustreError: 160761:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3993.574225] LustreError: 160761:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3994.741956] Lustre: server umount lustre-MDT0001 complete [ 3997.240993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4002.018143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4002.115910] LustreError: 164625:0:(llog.c:1655:llog_backup()) MGC192.168.201.145@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 4002.118875] Lustre: 164625:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.145@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 4011.870610] LustreError: 164630:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4023.137212] LustreError: 164630:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4023.148857] LustreError: 164630:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 4039.647786] LustreError: 164631:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4039.654773] LustreError: 164631:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [ 4062.815392] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 4062.818350] Lustre: 164625:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.145@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 4073.311613] LustreError: 164632:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4073.318630] LustreError: 164632:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 21 previous similar messages [ 4103.667114] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4103.672535] Lustre: Skipped 2 previous similar messages [ 4105.339691] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4108.772516] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4109.671827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4109.792477] LustreError: 147841:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000000709d806 x1875235960889600/t0(0) o250->MGC192.168.201.145@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4112.042511] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4115.427680] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4115.437446] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4116.217856] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4118.647639] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4119.382287] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4129.858204] Lustre: server umount lustre-MDT0000 complete [ 4130.786085] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4130.787192] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4130.796456] Lustre: Skipped 2 previous similar messages [ 4131.257302] LustreError: 160746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364734 with bad export cookie 9218570464430325361 [ 4131.260439] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4131.261553] LustreError: 160746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4131.396880] Lustre: server umount lustre-MDT0001 complete [ 4143.148809] Lustre: server umount lustre-OST0000 complete [ 4151.763908] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 4155.784479] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 4169.145399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4169.230747] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4169.243290] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4169.277865] Lustre: lustre-MDT0000: new disk, initializing [ 4169.304466] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4169.307075] Lustre: Skipped 1 previous similar message [ 4169.313173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4170.846326] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4175.418428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4175.473220] Lustre: 169696:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4175.599109] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4177.257393] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4181.461787] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4183.456620] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4185.962673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4186.085440] Lustre: 171663:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4188.283912] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4190.023332] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4190.081517] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4192.431214] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4193.944594] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4194.028481] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4195.437677] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4195.521804] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4200.417294] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4200.417350] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4200.419965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4200.419970] Lustre: Skipped 3 previous similar messages [ 4200.420371] LustreError: Skipped 1 previous similar message [ 4200.430687] Lustre: Skipped 2 previous similar messages [ 4202.323264] Lustre: server umount lustre-OST0000 complete [ 4209.816698] Lustre: server umount lustre-MDT0000 complete [ 4210.656047] LustreError: 170508:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4210.661104] LustreError: 170508:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 32 previous similar messages [ 4211.171155] LustreError: 169687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364814 with bad export cookie 9218570464430342161 [ 4211.173463] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4211.175102] LustreError: 169687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4211.300978] Lustre: server umount lustre-MDT0001 complete [ 4214.408436] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 12:00:16 (1788364816) [ 4216.862911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4218.707805] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4222.195509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4223.915212] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4226.959423] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4229.106758] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4231.736205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4234.112240] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4237.012119] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4246.420621] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 12:00:48 (1788364848) [ 4248.544429] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4248.551026] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4248.558805] Lustre: Skipped 2 previous similar messages [ 4248.561369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4248.565107] Lustre: Skipped 3 previous similar messages [ 4253.661421] Lustre: server umount lustre-OST0000 complete [ 4261.280991] Lustre: server umount lustre-MDT0000 complete [ 4262.696450] LustreError: 174052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364866 with bad export cookie 9218570464430343043 [ 4262.698849] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4262.700687] LustreError: 174052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4262.825651] Lustre: server umount lustre-MDT0001 complete [ 4267.036353] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 4268.258548] Key type lgssc unregistered [ 4268.399534] LNet: 177805:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4268.402934] LNetError: 177805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4268.412521] LNet: Removed LNI 192.168.201.145@tcp [ 4268.780198] Key type .llcrypt unregistered [ 4268.781288] Key type ._llcrypt unregistered [ 4277.373868] Key type ._llcrypt registered [ 4277.375095] Key type .llcrypt registered [ 4277.417557] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 4277.853823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4277.865190] alg: No test for adler32 (adler32-zlib) [ 4278.729958] Lustre: Lustre: Build Version: 2.17.56_2_ge76cbe5 [ 4278.832950] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 4280.423150] Key type lgssc registered [ 4280.834360] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4283.865570] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4283.874360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4299.295897] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4304.415336] LustreError: 179024:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.145@tcp: failed processing log, type 4: rc = -110 [ 4309.535584] LustreError: 179024:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 4309.540538] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 4309.543241] LustreError: MGC192.168.201.145@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 4309.548540] LustreError: 179024:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 4309.552639] LustreError: 179024:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 4309.555095] LustreError: 179024:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 4309.580255] Lustre: server umount lustre-OST0000 complete [ 4309.582603] LustreError: 179024:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 4312.899964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4314.064711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4315.635145] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4319.062740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4319.215264] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4320.834122] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4323.660726] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4325.574947] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4327.888057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4327.960446] LustreError: 181338:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 4327.964798] Lustre: 181338:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.201.145@tcp: invalid local config log lustre-OST0000: rc = -22 [ 4328.011424] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4330.284663] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4333.165149] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4334.562687] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4334.562987] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4334.563894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4334.567210] Lustre: Skipped 2 previous similar messages [ 4339.681037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4339.686343] Lustre: Skipped 4 previous similar messages [ 4344.801367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4344.804700] Lustre: Skipped 2 previous similar messages [ 4348.383108] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4348.439210] Lustre: server umount lustre-MDT0000 complete [ 4349.705641] LustreError: 179503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788364953 with bad export cookie 12227271136787448293 [ 4349.708866] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4349.709864] LustreError: 179503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4349.821809] Lustre: server umount lustre-MDT0001 complete [ 4361.631408] Lustre: server umount lustre-OST0000 complete [ 4364.874110] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 4368.386172] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 4380.759520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4380.844936] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4380.857352] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4380.893809] Lustre: lustre-MDT0000: new disk, initializing [ 4380.915665] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4380.922415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4382.380365] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4386.850633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4386.897686] Lustre: 184855:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4386.921975] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4386.924117] Lustre: Skipped 1 previous similar message [ 4386.961757] Lustre: lustre-MDT0001: new disk, initializing [ 4386.995173] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4387.009023] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4387.012485] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4388.504241] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4392.555079] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4394.634349] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4397.209934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4397.320616] Lustre: lustre-OST0000: new disk, initializing [ 4397.322915] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4397.325727] Lustre: 186825:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4397.361836] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4399.698593] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4403.879741] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4405.248948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4405.253731] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4405.286565] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4405.413915] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4405.488137] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4406.939480] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4407.017842] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4410.337133] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4410.338024] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4410.340865] LustreError: Skipped 1 previous similar message [ 4410.346340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4410.350288] Lustre: Skipped 1 previous similar message [ 4413.782191] Lustre: server umount lustre-OST0000 complete [ 4415.456225] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4415.457233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4415.460405] Lustre: Skipped 2 previous similar messages [ 4421.335564] Lustre: server umount lustre-MDT0000 complete [ 4422.669113] LustreError: 184850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788365025 with bad export cookie 12227271136787448993 [ 4422.672110] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4422.673254] LustreError: 184850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4422.790634] Lustre: server umount lustre-MDT0001 complete [ 4425.712949] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 12:03:48 (1788365028) [ 4431.856408] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 4432.870725] Key type lgssc unregistered [ 4432.998500] LNet: 189763:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4433.001517] LNetError: 189763:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4433.009477] LNet: Removed LNI 192.168.201.145@tcp [ 4433.336320] Key type .llcrypt unregistered [ 4433.337325] Key type ._llcrypt unregistered [ 4440.762438] Key type ._llcrypt registered [ 4440.763675] Key type .llcrypt registered [ 4440.803757] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 4441.192889] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4441.263758] alg: No test for adler32 (adler32-zlib) [ 4442.128890] Lustre: Lustre: Build Version: 2.17.56_2_ge76cbe5 [ 4442.226266] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 4443.815334] Key type lgssc registered [ 4444.200633] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4447.034142] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4447.040360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4448.170982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4449.502602] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4452.438768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4452.569342] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4453.919854] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4456.369516] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4458.322460] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4461.001720] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4461.069982] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4461.071639] LustreError: 192820:0:(llog.c:1696:llog_backup()) MGC192.168.201.145@tcp: failed to backup log lustre-OST0000: rc = -28 [ 4461.073769] Lustre: 192820:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.201.145@tcp: can't backup local config lustre-OST0000: rc = -28 [ 4461.106438] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4463.122787] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4465.569388] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4467.169923] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4467.172767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4467.175039] Lustre: Skipped 1 previous similar message [ 4472.288937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4472.291238] Lustre: Skipped 4 previous similar messages [ 4472.471795] Lustre: server umount lustre-MDT0000 complete [ 4473.711991] LustreError: 190939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788365077 with bad export cookie 510566201370096270 [ 4473.715367] LustreError: 190939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4473.715547] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4473.817988] Lustre: server umount lustre-MDT0001 complete [ 4485.333528] Lustre: server umount lustre-OST0000 complete [ 4488.422846] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 4491.610766] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 4503.927720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4504.010928] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4504.033830] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4504.073400] Lustre: lustre-MDT0000: new disk, initializing [ 4504.098234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4504.104168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4505.596049] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4509.994432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4510.038294] Lustre: 196337:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4510.056604] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4510.058138] Lustre: Skipped 1 previous similar message [ 4510.059895] LustreError: 197119:0:(llog.c:1696:llog_backup()) MGC192.168.201.145@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 4510.062709] LustreError: 197119:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 4510.064743] Lustre: 197119:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.145@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 4510.071777] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4510.073405] Lustre: Skipped 1 previous similar message [ 4510.108188] Lustre: lustre-MDT0001: new disk, initializing [ 4510.141700] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4510.154053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4510.158291] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4511.619860] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4515.342582] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4517.139250] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4519.339400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4519.426473] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4519.427779] Lustre: Skipped 1 previous similar message [ 4519.429229] LustreError: 198279:0:(llog.c:1696:llog_backup()) MGC192.168.201.145@tcp: failed to backup log lustre-OST0000: rc = -28 [ 4519.432775] LustreError: 198279:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 4519.434990] Lustre: 198279:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.145@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 4519.438548] Lustre: 198279:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 4519.447738] Lustre: lustre-OST0000: new disk, initializing [ 4519.449766] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4519.452418] Lustre: 198303:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4519.483811] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4521.431122] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4525.257901] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4526.793767] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4529.141461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4529.144377] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4529.169739] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4529.922645] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 4531.237500] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4531.308450] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4534.240754] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4534.242300] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4534.243201] LustreError: Skipped 1 previous similar message [ 4534.248721] Lustre: Skipped 1 previous similar message [ 4534.249428] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4534.254129] Lustre: Skipped 1 previous similar message [ 4538.011315] Lustre: server umount lustre-OST0000 complete [ 4539.360169] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4539.360574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4539.364324] Lustre: Skipped 2 previous similar messages [ 4539.367336] Lustre: Skipped 1 previous similar message [ 4544.480360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4544.482344] Lustre: Skipped 1 previous similar message [ 4545.365564] Lustre: server umount lustre-MDT0000 complete [ 4546.739963] LustreError: 196329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788365150 with bad export cookie 510566201370096970 [ 4546.743182] LustreError: 196329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4546.743613] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4546.861523] Lustre: server umount lustre-MDT0001 complete [ 4549.681357] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 12:05:52 (1788365152) [ 4555.397064] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 4558.890250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4559.020829] LustreError: 201449:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4559.029901] LustreError: 201449:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4559.035357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4560.502229] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4563.445250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4564.927886] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4565.954778] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4568.146639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4570.322469] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4573.201526] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4573.266551] Lustre: lustre-OST0001: new disk, initializing [ 4573.268304] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4573.271294] Lustre: 203754:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4575.320842] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4580.866805] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4581.377212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4581.380785] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4581.416548] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4586.606988] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4587.249903] Lustre: DEBUG MARKER: ADD OST3 [ 4588.900937] loop: module loaded [ 4588.918311] loop0: detected capacity change from 0 to 204800000 [ 4589.049184] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4589.053664] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4589.058302] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4591.182249] loop0: detected capacity change from 0 to 204800000 [ 4593.128963] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4593.200760] Lustre: lustre-OST0003: new disk, initializing [ 4593.202190] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 4593.204456] Lustre: 205483:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4593.241505] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 4593.243753] Lustre: Skipped 3 previous similar messages [ 4594.062980] Lustre: DEBUG MARKER: STOP OST3 [ 4594.730213] Lustre: Failing over lustre-OST0003 [ 4594.782671] Lustre: server umount lustre-OST0003 complete [ 4594.888234] LustreError: 202934:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4595.364217] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4599.310487] Lustre: DEBUG MARKER: START OST3 again [ 4600.161627] LustreError: 202934:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.201.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4600.168253] LustreError: 202934:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4601.593056] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4603.684549] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4606.954658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 4606.958266] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 4606.966941] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000401 [ 4617.339416] Lustre: Failing over lustre-OST0003 [ 4617.380731] Lustre: server umount lustre-OST0003 complete [ 4622.304210] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4622.307926] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4622.311635] LustreError: 202932:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4622.312158] Lustre: Skipped 1 previous similar message [ 4622.317272] LustreError: 202932:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4627.428420] LustreError: 202934:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4627.432792] LustreError: 202934:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4629.983814] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4629.987370] LustreError: Skipped 1 previous similar message [ 4629.989468] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4629.995744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4637.666573] LustreError: 202932:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4637.671179] LustreError: 202932:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4644.319104] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4644.380150] Lustre: server umount lustre-MDT0000 complete [ 4645.580352] LustreError: 201432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788365248 with bad export cookie 510566201370097838 [ 4645.584025] LustreError: 201432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4645.584213] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4647.904866] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4647.905831] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4647.909037] Lustre: Skipped 4 previous similar messages [ 4647.910757] Lustre: Skipped 12 previous similar messages [ 4651.826059] Lustre: server umount lustre-MDT0001 complete [ 4653.112073] Lustre: server umount lustre-OST0000 complete [ 4654.353220] Lustre: server umount lustre-OST0001 complete [ 4656.435862] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 4659.077736] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 4670.706556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4670.786079] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4670.796448] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4670.832626] Lustre: lustre-MDT0000: new disk, initializing [ 4670.852487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4670.854436] Lustre: Skipped 1 previous similar message [ 4670.859893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4672.224356] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4676.433947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4676.471579] Lustre: 210892:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4676.577248] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4677.933848] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4681.633875] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4683.666786] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4686.045085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4686.147263] Lustre: 212860:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4687.357662] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4687.360895] Lustre: Skipped 1 previous similar message [ 4687.363214] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4687.393587] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4688.129252] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4691.858657] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4693.276312] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4693.345050] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4694.705381] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4694.776228] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4695.446725] Lustre: server umount lustre-OST0000 complete [ 4697.055924] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4697.058273] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4697.062690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4697.064466] Lustre: Skipped 3 previous similar messages [ 4710.879121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4710.945573] Lustre: server umount lustre-MDT0000 complete [ 4712.188119] LustreError: 211687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788365315 with bad export cookie 510566201370100099 [ 4712.189754] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4712.192361] LustreError: 211687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4722.706639] Lustre: server umount lustre-MDT0001 complete [ 4724.023899] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 12:08:46 (1788365326) [ 4729.399683] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 4732.485255] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 4744.401610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4744.490384] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4744.499755] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4744.501460] Lustre: Skipped 3 previous similar messages [ 4744.536627] Lustre: lustre-MDT0000: new disk, initializing [ 4744.538740] Lustre: Skipped 2 previous similar messages [ 4744.566591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4746.113203] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4750.515509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4750.575211] Lustre: 217464:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4750.694245] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4752.152581] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4756.143326] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4758.329540] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4760.771712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4760.889035] Lustre: 219433:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4762.272915] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4763.040203] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4767.084178] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4768.718099] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4768.798531] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4770.303621] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4770.390466] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4772.834196] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4772.834679] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4772.835285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4772.835290] Lustre: Skipped 5 previous similar messages [ 4772.839044] Lustre: Skipped 2 previous similar messages [ 4772.841465] LustreError: Skipped 1 previous similar message [ 4777.195376] Lustre: server umount lustre-OST0000 complete [ 4781.535966] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4784.807621] Lustre: server umount lustre-MDT0000 complete [ 4786.178523] LustreError: 217457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788365389 with bad export cookie 510566201370100981 [ 4786.180406] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4786.183037] LustreError: 217457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4786.311045] Lustre: server umount lustre-MDT0001 complete [ 4789.549551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4789.720818] LustreError: 221643:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4789.727914] LustreError: 221643:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4791.143674] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4794.215538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4795.810241] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4798.408933] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4800.379246] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4802.857694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4803.020922] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4803.023402] Lustre: Skipped 7 previous similar messages [ 4805.234333] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 4808.123790] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4825.569635] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4831.758392] Lustre: server umount lustre-OST0000 complete [ 4849.119250] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4849.589095] Lustre: server umount lustre-MDT0000 complete [ 4851.168419] LustreError: 221639:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4851.179176] LustreError: 221639:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4852.526632] LustreError: 222378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788365455 with bad export cookie 510566201370101863 [ 4852.538276] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4853.027187] Lustre: server umount lustre-MDT0001 complete [ 4943.582704] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 4945.419381] Key type lgssc unregistered [ 4945.607729] LNet: 225329:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4945.612792] LNetError: 225329:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4946.662331] LNet: Removed LNI 192.168.201.145@tcp [ 4947.234165] Key type .llcrypt unregistered [ 4947.236223] Key type ._llcrypt unregistered [ 4963.826558] Key type ._llcrypt registered [ 4963.827527] Key type .llcrypt registered [ 4963.889712] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 4974.286269] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 4974.884098] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4974.938743] alg: No test for adler32 (adler32-zlib) [ 4975.869458] Lustre: Lustre: Build Version: 2.17.56_2_ge76cbe5 [ 4976.031937] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 4977.647152] Key type lgssc registered [ 4978.269874] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4999.532527] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4999.579305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5000.706395] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5000.735587] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5000.800306] Lustre: lustre-MDT0000: new disk, initializing [ 5000.872128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5000.884349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5003.641675] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5010.162712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5010.252362] Lustre: 228658:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5010.284357] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5010.286745] Lustre: Skipped 1 previous similar message [ 5010.340104] Lustre: lustre-MDT0001: new disk, initializing [ 5010.387055] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5010.412789] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5010.417627] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5013.158264] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5019.744814] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5023.828215] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5028.430442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5028.625274] Lustre: lustre-OST0000: new disk, initializing [ 5028.627272] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5028.630206] Lustre: 230625:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5028.696713] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5032.317751] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5036.582230] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5036.585920] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5036.664592] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5038.741402] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5042.108901] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5042.251276] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5045.057720] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5045.196924] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5046.756146] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5046.758201] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5046.764855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5046.767580] Lustre: Skipped 1 previous similar message [ 5051.872564] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5051.878021] Lustre: Skipped 1 previous similar message [ 5052.497720] Lustre: server umount lustre-OST0000 complete [ 5056.480150] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5056.485944] LustreError: Skipped 1 previous similar message [ 5056.488225] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5056.496745] Lustre: Skipped 1 previous similar message [ 5056.502134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5061.365823] Lustre: server umount lustre-MDT0000 complete [ 5063.408773] LustreError: 228650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788365666 with bad export cookie 18015041030604530798 [ 5063.409637] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5063.415188] LustreError: 228650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5063.584307] Lustre: server umount lustre-MDT0001 complete [ 5074.856937] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 5081.084724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5081.366910] LustreError: 233556:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5081.383369] LustreError: 233556:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5081.414196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5084.351859] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5086.692180] LustreError: 233557:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5090.092947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5090.398076] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5093.411743] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5095.308469] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5099.471658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5099.638963] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5103.322292] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5108.956953] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5109.061424] Lustre: lustre-OST0001: new disk, initializing [ 5109.063823] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5109.067828] Lustre: 235865:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5109.963586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5109.970975] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5110.050879] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5113.513958] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5121.773933] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5127.109315] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5135.842628] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 12:15:37 (1788365737) [ 5140.961966] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5140.973780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5140.976183] Lustre: Skipped 4 previous similar messages [ 5140.986954] Lustre: Skipped 4 previous similar messages [ 5143.544942] Lustre: server umount lustre-MDT0000 complete [ 5145.963403] LustreError: 233536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788365749 with bad export cookie 18015041030604531680 [ 5145.971104] LustreError: 233536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5145.972307] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5146.080830] LustreError: 233551:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5146.081746] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5146.088923] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5146.099678] LustreError: 233551:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5146.114237] Lustre: Skipped 1 previous similar message [ 5146.257319] Lustre: server umount lustre-MDT0001 complete [ 5154.987681] Lustre: server umount lustre-OST0000 complete [ 5157.502989] Lustre: server umount lustre-OST0001 complete [ 5162.156081] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 5168.949697] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 5193.046163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5193.176930] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5193.194522] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5193.246304] Lustre: lustre-MDT0000: new disk, initializing [ 5193.288575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5193.291064] Lustre: Skipped 1 previous similar message [ 5193.298377] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5195.996876] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5202.815136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5202.890044] Lustre: 239802:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5202.932493] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5202.935464] Lustre: Skipped 1 previous similar message [ 5202.985993] Lustre: lustre-MDT0001: new disk, initializing [ 5203.059614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5203.066586] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5205.486474] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5211.843618] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5215.125659] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5219.391835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5219.640461] Lustre: lustre-OST0000: new disk, initializing [ 5219.645993] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5219.653838] Lustre: 241770:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5220.891114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5220.898359] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5220.978427] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5223.751030] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5229.870699] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5232.651648] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5232.790712] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5235.694981] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5235.814624] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5241.313070] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5241.316642] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5241.322814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5241.332083] Lustre: Skipped 1 previous similar message [ 5241.341626] Lustre: Skipped 1 previous similar message [ 5243.070134] Lustre: server umount lustre-OST0000 complete [ 5251.827314] Lustre: server umount lustre-MDT0000 complete [ 5254.357432] LustreError: 240597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788365857 with bad export cookie 18015041030604533815 [ 5254.360941] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5254.367832] LustreError: 240597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5254.578080] Lustre: server umount lustre-MDT0001 complete [ 5259.989780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5260.292315] LustreError: 243983:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5260.334786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5260.337294] Lustre: Skipped 2 previous similar messages [ 5263.246256] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5265.378240] LustreError: 243984:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5268.987934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5272.559898] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5277.514983] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5281.402527] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5286.309909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5290.322612] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5295.483387] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5443.554194] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5443.555529] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5443.557876] LustreError: Skipped 1 previous similar message [ 5443.559732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5443.565426] Lustre: Skipped 3 previous similar messages [ 5443.575152] Lustre: Skipped 6 previous similar messages [ 5445.295659] Lustre: server umount lustre-OST0000 complete [ 5448.680956] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5453.624235] Lustre: server umount lustre-MDT0000 complete [ 5453.793103] LustreError: 243983:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5453.802473] LustreError: 243983:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5455.537185] LustreError: 243963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788366058 with bad export cookie 18015041030604534697 [ 5455.542092] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5455.546299] LustreError: 243963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5455.727480] Lustre: server umount lustre-MDT0001 complete [ 5566.769269] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 5569.964383] Key type lgssc unregistered [ 5570.444140] LNet: 247590:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5570.473073] LNetError: 247590:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5570.492753] LNet: Removed LNI 192.168.201.145@tcp [ 5571.509589] Key type .llcrypt unregistered [ 5571.512538] Key type ._llcrypt unregistered [ 5598.677837] Key type ._llcrypt registered [ 5598.680891] Key type .llcrypt registered [ 5598.812472] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 5613.376544] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 5614.612482] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5614.688751] alg: No test for adler32 (adler32-zlib) [ 5615.690361] Lustre: Lustre: Build Version: 2.17.56_2_ge76cbe5 [ 5615.874194] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 5617.527227] Key type lgssc registered [ 5618.379211] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5655.170701] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 5663.102961] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5663.129756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5664.382643] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5664.429613] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5664.504651] Lustre: lustre-MDT0000: new disk, initializing [ 5664.596664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5664.609480] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5668.418825] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5677.987400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5678.113850] Lustre: 251661:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5678.192236] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5678.196603] Lustre: Skipped 1 previous similar message [ 5678.262317] Lustre: lustre-MDT0001: new disk, initializing [ 5678.358810] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5678.398243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5678.412799] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5682.712867] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5687.809983] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5694.589936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5694.998328] Lustre: lustre-OST0000: new disk, initializing [ 5695.005574] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5695.018050] Lustre: 253290:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5695.151048] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5697.519092] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5697.525797] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5697.650716] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5701.551587] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5712.373931] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5712.583638] Lustre: lustre-OST0001: new disk, initializing [ 5712.587359] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5712.596676] Lustre: 254161:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5712.705361] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5718.734495] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5720.635962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5720.646637] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5720.740149] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5729.616126] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5735.670876] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5746.667700] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5746.671901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5746.682104] Lustre: Skipped 1 previous similar message [ 5751.777513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5751.781955] Lustre: Skipped 3 previous similar messages [ 5752.337856] Lustre: server umount lustre-MDT0000 complete [ 5755.943544] LustreError: 251653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788366359 with bad export cookie 10754797769090508577 [ 5755.952558] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5755.964753] LustreError: 251653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5756.271600] Lustre: server umount lustre-MDT0001 complete [ 5770.572442] Lustre: server umount lustre-OST0000 complete [ 5772.767549] Lustre: 249170:0:(client.c:2493:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788366360/real 1788366360] req@ffff9182c338fb80 x1875237896529024/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788366376 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 5772.792527] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5772.799900] Lustre: Skipped 2 previous similar messages [ 5774.227765] Lustre: server umount lustre-OST0001 complete [ 5785.322150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5792.927364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5799.670596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5808.006182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5823.103736] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 5831.770347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5831.783388] Lustre: lustre-MDT0000: reset Object Index mappings [ 5832.077131] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5835.794748] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5837.282563] LustreError: 259167:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5837.296247] LustreError: 259167:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5842.405897] LustreError: 259166:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5842.979813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5843.006212] Lustre: lustre-MDT0001: reset Object Index mappings [ 5843.488265] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5847.695595] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5850.311698] Lustre: 260307:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5856.219858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5857.577613] LustreError: 260662:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5857.607528] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5862.495413] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5862.890770] LustreError: 260662:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5868.007993] LustreError: 260662:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5868.021304] LustreError: 260662:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5869.761715] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5869.968257] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5869.971330] Lustre: Skipped 1 previous similar message [ 5875.216320] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 5875.897467] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 5882.933576] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5885.489186] Lustre: 262176:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5889.504290] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5889.508576] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5889.528072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5889.531298] Lustre: Skipped 3 previous similar messages [ 5895.176400] Lustre: server umount lustre-MDT0000 complete [ 5895.649618] LustreError: 261047:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5895.668092] LustreError: 261047:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5897.883328] LustreError: 259148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788366501 with bad export cookie 10754797769090515367 [ 5897.905143] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5898.178331] Lustre: server umount lustre-MDT0001 complete [ 5912.082121] Lustre: server umount lustre-OST0000 complete [ 5924.952687] Lustre: server umount lustre-OST0001 complete [ 5942.558964] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 12:29:04 (1788366544) [ 5958.781746] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 5967.692990] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 6001.938950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6002.239307] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6002.269671] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6002.373328] Lustre: lustre-MDT0000: new disk, initializing [ 6002.472214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6002.488664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6007.286480] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6017.992153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6018.125533] Lustre: 266434:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6018.175881] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6018.179113] Lustre: Skipped 1 previous similar message [ 6018.262685] Lustre: lustre-MDT0001: new disk, initializing [ 6018.413708] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6018.428228] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6022.874562] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6032.513658] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6037.546770] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6043.819970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6044.105907] Lustre: lustre-OST0000: new disk, initializing [ 6044.110831] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6044.119977] Lustre: 268402:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6045.520969] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6045.534663] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6045.602310] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6049.862203] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6058.428688] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6063.202245] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6063.484846] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6067.476347] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6067.668995] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6071.270520] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6071.281801] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6071.293746] Lustre: Skipped 3 previous similar messages [ 6071.298308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6071.307511] Lustre: Skipped 5 previous similar messages [ 6075.700117] Lustre: server umount lustre-OST0000 complete [ 6079.968533] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6079.974613] LustreError: Skipped 1 previous similar message [ 6079.982817] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6080.001871] Lustre: Skipped 1 previous similar message [ 6080.007161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6085.548921] Lustre: server umount lustre-MDT0000 complete [ 6086.624959] LustreError: 267949:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6086.655437] LustreError: 267949:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6088.898045] LustreError: 266427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788366692 with bad export cookie 10754797769090517446 [ 6088.901736] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6088.910677] LustreError: 266427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6089.237630] Lustre: server umount lustre-MDT0001 complete [ 6107.159765] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 6116.749965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6117.240014] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6117.241967] Lustre: Skipped 2 previous similar messages [ 6121.790363] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6122.467693] LustreError: 271337:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6122.480157] LustreError: 271337:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6129.148297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6133.382834] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6135.843741] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6141.512662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6147.648266] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6154.477478] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6154.589690] Lustre: lustre-OST0001: new disk, initializing [ 6154.591773] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6154.594404] Lustre: 273639:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6155.725578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6155.737830] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6155.830693] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6159.474297] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6168.204470] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6171.445366] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6175.806933] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 6175.834972] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 6180.837511] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6180.845846] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6180.857730] Lustre: Skipped 1 previous similar message [ 6180.861464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6180.870027] Lustre: Skipped 2 previous similar messages [ 6182.848674] Lustre: server umount lustre-MDT0000 complete [ 6186.133293] LustreError: 271318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788366789 with bad export cookie 10754797769090518335 [ 6186.135392] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6186.141897] LustreError: 271318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6186.578935] Lustre: server umount lustre-MDT0001 complete [ 6200.119759] Lustre: server umount lustre-OST0000 complete [ 6213.827646] Lustre: server umount lustre-OST0001 complete [ 6226.582710] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 6233.992978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6234.336331] LustreError: 276694:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6234.356425] LustreError: 276694:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6237.913653] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6244.353325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6248.167866] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6250.393853] Lustre: 277835:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6255.792415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6256.013402] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6256.016932] Lustre: Skipped 5 previous similar messages [ 6259.156803] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 6259.839669] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6265.641639] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6268.854922] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 6270.926386] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6277.021993] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6279.927613] Lustre: 279714:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6283.619628] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 6283.622885] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 6290.864808] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 12:34:52 (1788366892) [ 6304.742901] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6304.752149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6304.760538] Lustre: Skipped 5 previous similar messages [ 6304.765172] Lustre: Skipped 1 previous similar message [ 6312.839435] Lustre: server umount lustre-MDT0000 complete [ 6315.676819] LustreError: 277837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788366918 with bad export cookie 10754797769090524642 [ 6315.697456] LustreError: 277837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6315.705859] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6322.011881] Lustre: server umount lustre-MDT0001 complete [ 6331.212918] Lustre: server umount lustre-OST0000 complete [ 6340.235710] Lustre: server umount lustre-OST0001 complete [ 6345.269837] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 6352.084218] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 6385.839323] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 6393.258980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6393.454496] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6393.489041] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6393.542619] Lustre: lustre-MDT0000: new disk, initializing [ 6393.620475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6396.383978] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6403.230167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6403.317981] Lustre: 283948:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6403.476941] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6406.611355] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6415.965351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6416.180961] Lustre: 285580:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6418.088796] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6421.501674] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6429.125717] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6429.231213] Lustre: 286451:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6431.034470] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6433.230375] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6440.366129] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6442.821198] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6442.826318] Lustre: Skipped 1 previous similar message [ 6452.445569] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 12:37:34 (1788367054) [ 6454.756150] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6454.762340] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6454.774673] Lustre: Skipped 3 previous similar messages [ 6454.779628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6454.784468] Lustre: Skipped 7 previous similar messages [ 6460.171423] Lustre: server umount lustre-MDT0000 complete [ 6461.926174] LustreError: 283955:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6461.942259] LustreError: 283955:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 6463.073026] LustreError: 283940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788367066 with bad export cookie 10754797769090532531 [ 6463.077232] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6463.079210] LustreError: 283940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6463.302488] Lustre: server umount lustre-MDT0001 complete [ 6476.352290] Lustre: server umount lustre-OST0000 complete [ 6489.411864] Lustre: server umount lustre-OST0001 complete [ 6502.123038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6507.885667] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6519.910159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6520.579542] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6520.592130] Lustre: Skipped 6 previous similar messages [ 6527.145786] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6538.509412] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6546.095268] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6553.415293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6560.128785] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6571.052942] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6595.464769] loop1: detected capacity change from 0 to 204800000 [ 6595.691164] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6595.707296] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6595.725157] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6600.830656] loop1: detected capacity change from 0 to 204800000 [ 6607.975621] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6608.075542] LustreError: 288682:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs22447: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 6608.099991] LustreError: 288682:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs22447 [ 6608.111779] LustreError: 288682:0:(mgs_handler.c:517:mgs_target_reg()) fs22447-MDT0001: Error checking target index: rc = -13 [ 6608.121395] LustreError: fs22447-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 6608.128500] LustreError: 292416:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 6608.134435] LustreError: 292416:0:(tgt_mount.c:2129:server_put_super()) no obd fs22447-MDT0001 [ 6608.139531] LustreError: 292416:0:(tgt_mount.c:116:server_deregister_mount()) fs22447-MDT0001 not registered [ 6608.177082] Lustre: server umount fs22447-MDT0001 complete [ 6608.180489] LustreError: 292416:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 6617.419660] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6617.729466] Lustre: Setting parameter fs22447-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs22447-MDT0001 [ 6617.849105] Lustre: srv-fs22447-MDT0001: No data found on store. Initialize space. [ 6617.856851] Lustre: Skipped 4 previous similar messages [ 6618.027636] Lustre: fs22447-MDT0001: new disk, initializing [ 6618.039651] Lustre: Skipped 3 previous similar messages [ 6618.269565] Lustre: srv-fs22447-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 6618.285375] Lustre: 292882:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs22447-MDT0001-mdtlov: can't create new llog: rc = -115 [ 6618.302839] LustreError: 292882:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs22447-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 6623.181638] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6630.200685] Lustre: server umount fs22447-MDT0001 complete [ 6642.161989] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [ 6644.165669] loop1: detected capacity change from 0 to 204800000 [ 6650.226282] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6650.637474] Lustre: 294450:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs22447-MDT0001-mdtlov: can't create new llog: rc = -115 [ 6650.659137] LustreError: 294450:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs22447-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 6655.667594] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6660.722724] Lustre: server umount fs22447-MDT0001 complete [ 6672.876605] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6672.892062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6672.892146] Lustre: Skipped 5 previous similar messages [ 6672.902709] Lustre: Skipped 6 previous similar messages [ 6685.151221] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6685.446258] Lustre: server umount lustre-MDT0000 complete [ 6693.973061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6694.142452] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6699.010540] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6699.508869] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6707.126952] loop1: detected capacity change from 0 to 204800000 [ 6713.086707] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6713.739030] Lustre: 296896:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs22447-MDT0001-mdtlov: can't create new llog: rc = -115 [ 6713.745588] LustreError: 296896:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs22447-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 6718.261551] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6719.974952] LustreError: 290525:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6719.987566] LustreError: 290525:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 97 previous similar messages [ 6723.371039] Lustre: server umount fs22447-MDT0001 complete [ 6751.541878] Lustre: server umount lustre-OST0000 complete [ 6755.808742] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6755.824026] LustreError: Skipped 1 previous similar message [ 6761.596878] Lustre: server umount lustre-MDT0000 complete [ 6765.769608] LustreError: 288675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788367369 with bad export cookie 10754797769090535212 [ 6765.777995] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6765.798741] LustreError: 288675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6766.322104] Lustre: server umount lustre-MDT0001 complete [ 6777.469352] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 6780.506774] Key type lgssc unregistered [ 6780.824825] LNet: 299240:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6780.838834] LNetError: 299240:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6780.859492] LNet: Removed LNI 192.168.201.145@tcp [ 6781.839564] Key type .llcrypt unregistered [ 6781.842598] Key type ._llcrypt unregistered [ 6800.084541] Key type ._llcrypt registered [ 6800.089503] Key type .llcrypt registered [ 6800.200712] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 6801.058418] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6801.071749] alg: No test for adler32 (adler32-zlib) [ 6802.374573] Lustre: Lustre: Build Version: 2.17.56_2_ge76cbe5 [ 6802.618993] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 6804.279185] Key type lgssc registered [ 6805.198733] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6811.684243] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6811.700329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6813.195805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6817.840261] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6823.393816] LustreError: 300511:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6823.426931] LustreError: 300511:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6826.111357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6826.406374] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6830.038332] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6838.404535] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6843.715959] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6850.137418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6850.558917] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6852.583812] LustreError: 302340:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6855.984618] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6857.702800] LustreError: 302339:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6857.723904] LustreError: 302339:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6862.415143] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6862.821619] LustreError: 302483:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6862.837652] LustreError: 302483:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6867.944205] LustreError: 302340:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6867.971884] LustreError: 302340:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6878.188495] LustreError: 302483:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6878.200852] LustreError: 302483:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6878.213352] loop1: detected capacity change from 0 to 204800000 [ 6878.426053] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6878.445975] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6878.456549] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6882.739341] loop1: detected capacity change from 0 to 204800000 [ 6887.892354] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6888.027818] Lustre: fs6221-OST0000: new disk, initializing [ 6888.031286] Lustre: srv-fs6221-OST0000: No data found on store. Initialize space. [ 6888.037759] Lustre: 304245:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6888.120549] Lustre: fs6221-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6893.314632] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6894.959571] LustreError: 302341:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6894.988122] LustreError: 302341:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 6900.783178] Lustre: server umount fs6221-OST0000 complete [ 6910.927835] loop1: detected capacity change from 0 to 204800000 [ 6915.652252] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6915.820966] Lustre: fs6221-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6920.059777] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6925.878049] loop2: detected capacity change from 0 to 204800000 [ 6926.027781] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6926.034821] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6926.042168] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6929.390317] LustreError: 302340:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6929.408585] LustreError: 302340:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 19 previous similar messages [ 6929.850168] loop2: detected capacity change from 0 to 204800000 [ 6934.138949] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6934.197134] LustreError: fs6221-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 6934.204488] LustreError: fs6221-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 6934.208338] LustreError: 307284:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 6934.211710] LustreError: 307284:0:(tgt_mount.c:2129:server_put_super()) no obd fs6221-OST0001 [ 6934.214039] LustreError: 307284:0:(tgt_mount.c:116:server_deregister_mount()) fs6221-OST0001 not registered [ 6934.230333] Lustre: server umount fs6221-OST0001 complete [ 6934.231907] LustreError: 307284:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 6939.619778] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [ 6941.719085] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6941.817916] Lustre: fs6221-OST0001: new disk, initializing [ 6941.820762] Lustre: srv-fs6221-OST0001: No data found on store. Initialize space. [ 6941.825804] Lustre: 307834:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6941.887917] Lustre: fs6221-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6946.961619] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6952.980497] Lustre: server umount fs6221-OST0000 complete [ 6959.088080] Lustre: server umount fs6221-OST0001 complete [ 6966.321398] loop1: detected capacity change from 0 to 204800000 [ 6969.736977] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6969.862667] Lustre: fs6221-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6974.011291] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6978.203687] loop2: detected capacity change from 0 to 204800000 [ 6981.740759] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6985.915809] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 6991.850093] Lustre: server umount fs6221-OST0000 complete [ 6995.941839] LustreError: 302341:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6995.958376] LustreError: 302341:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 38 previous similar messages [ 6998.010934] Lustre: server umount fs6221-OST0001 complete [ 7006.182404] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7006.188434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7006.189677] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7006.200624] LustreError: Skipped 2 previous similar messages [ 7011.116473] Lustre: server umount lustre-OST0000 complete [ 7015.905224] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7015.911568] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7015.932825] Lustre: Skipped 1 previous similar message [ 7015.937768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7015.945455] Lustre: Skipped 1 previous similar message [ 7021.537728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7021.544686] Lustre: Skipped 3 previous similar messages [ 7026.658121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7026.662903] Lustre: Skipped 1 previous similar message [ 7028.194269] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7028.296980] Lustre: server umount lustre-MDT0000 complete [ 7030.756318] LustreError: 300490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788367634 with bad export cookie 14330966683360032284 [ 7030.761105] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7030.763187] LustreError: 300490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7030.931657] Lustre: server umount lustre-MDT0001 complete [ 7037.502572] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 7039.560488] Key type lgssc unregistered [ 7039.791775] LNet: 313754:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7039.798305] LNetError: 313754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7039.807498] LNet: Removed LNI 192.168.201.145@tcp [ 7040.559114] Key type .llcrypt unregistered [ 7040.561062] Key type ._llcrypt unregistered [ 7061.259832] Key type ._llcrypt registered [ 7061.261453] Key type .llcrypt registered [ 7061.329725] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 7072.118180] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 7072.880218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7072.897686] alg: No test for adler32 (adler32-zlib) [ 7073.867989] Lustre: Lustre: Build Version: 2.17.56_2_ge76cbe5 [ 7074.017535] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 7075.658142] Key type lgssc registered [ 7076.407680] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7102.294171] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7102.334594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7103.504468] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7103.531701] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7103.589913] Lustre: lustre-MDT0000: new disk, initializing [ 7103.656153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7103.665955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7106.895722] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7114.941025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7115.015668] Lustre: 317590:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7115.051695] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7115.054125] Lustre: Skipped 1 previous similar message [ 7115.112318] Lustre: lustre-MDT0001: new disk, initializing [ 7115.193881] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7115.213691] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7115.223829] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7118.217814] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7125.492867] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7129.768502] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7134.617158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7134.775417] Lustre: lustre-OST0000: new disk, initializing [ 7134.782403] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7134.787384] Lustre: 319560:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7134.871539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7139.106384] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7142.447416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7142.458797] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7142.549065] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7146.233392] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7147.986838] Lustre: Mounted lustre-client [ 7148.805224] Lustre: Unmounted lustre-client [ 7158.253335] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7158.253358] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7158.260502] LustreError: Skipped 1 previous similar message [ 7158.264227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7158.293507] Lustre: Skipped 1 previous similar message [ 7162.098462] Lustre: server umount lustre-OST0000 complete [ 7166.443935] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7166.449791] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7166.465478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7166.470639] Lustre: Skipped 1 previous similar message [ 7168.485075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7168.491534] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7170.858366] Lustre: server umount lustre-MDT0000 complete [ 7173.559799] LustreError: 318385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788367776 with bad export cookie 16543035456996921767 [ 7173.567229] LustreError: 318385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7173.568346] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7173.895834] Lustre: server umount lustre-MDT0001 complete [ 7181.747986] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 7183.789087] Key type lgssc unregistered [ 7204.511604] LNet: 321717:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7204.518033] LNetError: 321717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7280.310447] LNet: Removed LNI 192.168.201.145@tcp [ 7280.879089] Key type .llcrypt unregistered [ 7280.884768] Key type ._llcrypt unregistered [ 7292.430763] Key type ._llcrypt registered [ 7292.432365] Key type .llcrypt registered [ 7292.503828] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 7302.501905] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 7303.322704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7303.368876] alg: No test for adler32 (adler32-zlib) [ 7304.336575] Lustre: Lustre: Build Version: 2.17.56_2_ge76cbe5 [ 7304.511335] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 7306.175154] Key type lgssc registered [ 7306.852732] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7325.589179] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 12:52:07 (1788367927) [ 7329.990090] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7330.004043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7331.174364] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7331.193851] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7331.238182] Lustre: lustre-MDT0000: new disk, initializing [ 7331.287114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7331.299378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7333.963491] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7341.877193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7341.959543] Lustre: 324918:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7342.002934] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7342.006303] Lustre: Skipped 1 previous similar message [ 7342.055511] Lustre: lustre-MDT0001: new disk, initializing [ 7342.142119] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7342.180835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7342.188702] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7344.674918] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7350.667900] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7354.402935] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7359.003225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7359.210189] Lustre: lustre-OST0000: new disk, initializing [ 7359.212391] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7359.218389] Lustre: 326883:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7359.279082] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7363.259847] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7368.213237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7368.221746] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7368.270531] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7369.204392] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7378.402380] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7378.402661] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7378.405326] LustreError: Skipped 1 previous similar message [ 7378.414081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7378.418086] Lustre: Skipped 1 previous similar message [ 7383.521380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7383.526456] Lustre: Skipped 1 previous similar message [ 7384.344188] Lustre: server umount lustre-OST0000 complete [ 7388.129732] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7388.135621] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7388.152615] Lustre: Skipped 1 previous similar message [ 7388.161447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7392.594639] Lustre: server umount lustre-MDT0000 complete [ 7395.365979] LustreError: 324909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788367998 with bad export cookie 13770302849742924813 [ 7395.372119] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7395.374360] LustreError: 324909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7395.672826] Lustre: server umount lustre-MDT0001 complete [ 7403.291183] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 7405.179867] Key type lgssc unregistered [ 7405.391742] LNet: 328787:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7405.396823] LNetError: 328787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7405.416408] LNet: Removed LNI 192.168.201.145@tcp [ 7406.023915] Key type .llcrypt unregistered [ 7406.025203] Key type ._llcrypt unregistered [ 7419.713984] Key type ._llcrypt registered [ 7419.717150] Key type .llcrypt registered [ 7419.782269] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 7420.380630] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7420.529952] alg: No test for adler32 (adler32-zlib) [ 7421.534737] Lustre: Lustre: Build Version: 2.17.56_2_ge76cbe5 [ 7421.735300] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 7423.343172] Key type lgssc registered [ 7424.013967] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7429.471830] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7429.485794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7430.818086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7434.287127] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7440.036859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7440.287394] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7443.003864] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7447.696803] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7451.449165] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7456.182370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7456.389469] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7460.248863] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7465.976632] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7473.104589] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7474.090457] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7478.129689] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.45@tcp (stopping) [ 7481.313523] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7481.313632] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7481.321213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7481.324397] Lustre: Skipped 1 previous similar message [ 7482.575970] Lustre: server umount lustre-MDT0000 complete [ 7483.232803] LustreError: 330037:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7483.242663] LustreError: 330037:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7484.586633] LustreError: 330020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788368087 with bad export cookie 14451547552454781398 [ 7484.594646] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7486.432586] LustreError: 330035:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7486.432608] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7486.432616] Lustre: Skipped 1 previous similar message [ 7486.451964] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7486.455973] Lustre: Skipped 2 previous similar messages [ 7488.351656] LustreError: 330036:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7489.440887] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7489.446057] Lustre: Skipped 1 previous similar message [ 7490.902347] Lustre: server umount lustre-MDT0001 complete [ 7499.463817] Lustre: server umount lustre-OST0000 complete [ 7504.662749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7507.017895] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7508.792477] Lustre: 334218:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.145@tcp [ 7509.991373] Lustre: server umount MGS complete [ 7515.096934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7515.352343] LustreError: 334789:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7515.370132] LustreError: 334789:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7515.390580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7518.030895] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7520.737821] LustreError: 334790:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7523.317710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7523.556972] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7526.051287] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7531.698920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7531.881852] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7535.791153] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7540.714226] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7541.756873] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [ 7545.146949] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7546.126712] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7550.439946] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7550.440618] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7550.445144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7550.454680] Lustre: Skipped 1 previous similar message [ 7554.728750] Lustre: server umount lustre-OST0000 complete [ 7559.648284] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7559.651909] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7559.667946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7559.673411] Lustre: Skipped 1 previous similar message [ 7563.354448] Lustre: server umount lustre-MDT0000 complete [ 7565.709924] LustreError: 334770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788368169 with bad export cookie 14451547552454782238 [ 7565.715890] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7565.719617] LustreError: 334770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7565.934850] Lustre: server umount lustre-MDT0001 complete [ 7572.316344] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 7574.007984] Key type lgssc unregistered [ 7574.217807] LNet: 338218:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7574.225215] LNetError: 338218:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7574.244477] LNet: Removed LNI 192.168.201.145@tcp [ 7574.861077] Key type .llcrypt unregistered [ 7574.863899] Key type ._llcrypt unregistered [ 7584.761625] Key type ._llcrypt registered [ 7584.763374] Key type .llcrypt registered [ 7584.829569] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 7594.245744] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 7594.781535] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7594.795199] alg: No test for adler32 (adler32-zlib) [ 7595.753857] Lustre: Lustre: Build Version: 2.17.56_2_ge76cbe5 [ 7595.906299] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 7597.527383] Key type lgssc registered [ 7598.200946] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7620.337310] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7620.353412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7621.559822] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7621.587958] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7621.646749] Lustre: lustre-MDT0000: new disk, initializing [ 7621.712505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7621.725104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7624.013205] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7630.714664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7630.812479] Lustre: 341358:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7630.855640] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7630.858775] Lustre: Skipped 1 previous similar message [ 7630.906289] Lustre: lustre-MDT0001: new disk, initializing [ 7630.963645] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7630.994082] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7630.999403] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7634.014794] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7640.121503] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7643.443631] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7648.084702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7648.272596] Lustre: lustre-OST0000: new disk, initializing [ 7648.274623] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7648.277841] Lustre: 343327:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7648.346239] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7652.041485] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7656.990969] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7656.997138] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7657.078617] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7658.892615] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7662.244882] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7662.387342] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7665.420885] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7665.579927] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7667.170349] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7667.172695] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7667.185115] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7672.295187] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7672.298721] Lustre: Skipped 1 previous similar message [ 7673.054166] Lustre: server umount lustre-OST0000 complete [ 7677.408494] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7677.412285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7677.426439] Lustre: Skipped 2 previous similar messages [ 7677.435360] Lustre: Skipped 2 previous similar messages [ 7682.531051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7682.540346] Lustre: Skipped 1 previous similar message [ 7687.648412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7687.650741] Lustre: Skipped 1 previous similar message [ 7690.207255] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7690.321547] Lustre: server umount lustre-MDT0000 complete [ 7692.437319] LustreError: 342160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788368295 with bad export cookie 4233692204225666484 [ 7692.439532] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7692.444379] LustreError: 342160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7692.613416] Lustre: server umount lustre-MDT0001 complete [ 7697.575842] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 12:58:19 (1788368299) [ 7706.495637] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 7712.165627] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 7743.145547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7743.366180] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7743.402674] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7743.496211] Lustre: lustre-MDT0000: new disk, initializing [ 7743.588659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7743.609689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7748.135568] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7761.461704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7761.567754] Lustre: 348078:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7761.655157] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7761.658361] Lustre: Skipped 1 previous similar message [ 7761.708177] Lustre: lustre-MDT0001: new disk, initializing [ 7762.104880] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7762.138432] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7762.148793] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7767.316352] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7779.151526] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7787.059847] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7798.660930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7799.130295] Lustre: lustre-OST0000: new disk, initializing [ 7799.134532] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7799.145623] Lustre: 350046:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7799.354822] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7805.004627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7805.012087] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7805.104068] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7806.106322] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 7816.584777] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7846.344860] Lustre: server umount lustre-OST0000 complete [ 7849.439921] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7849.447527] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7849.453148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7851.489543] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7859.743965] Lustre: server umount lustre-MDT0000 complete [ 7862.689782] Lustre: 339581:0:(client.c:2493:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788368449/real 1788368449] req@00000000c00fecc8 x1875239972721920/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788368465 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7862.715848] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7863.005868] LustreError: 348069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788368466 with bad export cookie 4233692204225667366 [ 7863.013134] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7863.027916] LustreError: 348069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7863.536522] Lustre: server umount lustre-MDT0001 complete [ 7873.251000] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 7875.502563] Key type lgssc unregistered [ 7875.846754] LNet: 352207:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7875.851293] LNetError: 352207:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7875.870663] LNet: Removed LNI 192.168.201.145@tcp [ 7876.653535] Key type .llcrypt unregistered [ 7876.655317] Key type ._llcrypt unregistered [ 7946.961843] Key type ._llcrypt registered [ 7946.964144] Key type .llcrypt registered [ 7947.036683] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 7947.193400] Key type .llcrypt unregistered [ 7947.194493] Key type ._llcrypt unregistered [ 8005.812585] Key type ._llcrypt registered [ 8005.814474] Key type .llcrypt registered [ 8005.904340] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 8006.148691] Key type .llcrypt unregistered [ 8006.154809] Key type ._llcrypt unregistered [ 8029.215846] Key type ._llcrypt registered [ 8029.221593] Key type .llcrypt registered [ 8029.304649] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 8029.530285] Key type .llcrypt unregistered [ 8029.532807] Key type ._llcrypt unregistered [ 8054.875832] Key type ._llcrypt registered [ 8054.877940] Key type .llcrypt registered [ 8054.976738] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 8055.208492] Key type .llcrypt unregistered [ 8055.211840] Key type ._llcrypt unregistered [ 8110.879599] Key type ._llcrypt registered [ 8110.881506] Key type .llcrypt registered [ 8111.004085] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 8111.261894] Key type .llcrypt unregistered [ 8111.263925] Key type ._llcrypt unregistered [ 8133.997520] Key type ._llcrypt registered [ 8133.999749] Key type .llcrypt registered [ 8134.071590] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 8134.271506] Key type .llcrypt unregistered [ 8134.274182] Key type ._llcrypt unregistered [ 8155.240048] Key type ._llcrypt registered [ 8155.241445] Key type .llcrypt registered [ 8155.336126] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 8155.557573] Key type .llcrypt unregistered [ 8155.559233] Key type ._llcrypt unregistered [ 8176.789802] Key type ._llcrypt registered [ 8176.792519] Key type .llcrypt registered [ 8176.880071] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 8177.132616] Key type .llcrypt unregistered [ 8177.137661] Key type ._llcrypt unregistered [ 8200.263590] Key type ._llcrypt registered [ 8200.265470] Key type .llcrypt registered [ 8200.351779] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 8200.563447] Key type .llcrypt unregistered [ 8200.565194] Key type ._llcrypt unregistered [ 8220.342115] Key type ._llcrypt registered [ 8220.344096] Key type .llcrypt registered [ 8220.434628] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 8220.618343] Key type .llcrypt unregistered [ 8220.623883] Key type ._llcrypt unregistered [ 8262.112532] Key type ._llcrypt registered [ 8262.114299] Key type .llcrypt registered [ 8262.182530] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 8262.376604] Key type .llcrypt unregistered [ 8262.380849] Key type ._llcrypt unregistered [ 8284.870693] Key type ._llcrypt registered [ 8284.875168] Key type .llcrypt registered [ 8284.972426] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 8285.189321] Key type .llcrypt unregistered [ 8285.193188] Key type ._llcrypt unregistered [ 8308.263776] Key type ._llcrypt registered [ 8308.265707] Key type .llcrypt registered [ 8308.354927] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 8308.585803] Key type .llcrypt unregistered [ 8308.590461] Key type ._llcrypt unregistered [ 8348.678314] Key type ._llcrypt registered [ 8348.680315] Key type .llcrypt registered [ 8348.768501] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 8359.297430] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 8359.810477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8359.828540] alg: No test for adler32 (adler32-zlib) [ 8360.833565] Lustre: Lustre: Build Version: 2.17.56_2_ge76cbe5 [ 8361.009369] LNet: Added LNI 192.168.201.145@tcp [8/256/0/180] [ 8362.647163] Key type lgssc registered [ 8363.408618] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8386.038029] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8386.062796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8387.231737] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 8387.244742] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 8387.308966] Lustre: tst250fs-MDT0000: new disk, initializing [ 8387.369690] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8387.384670] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8390.423881] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 8397.860814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8397.955832] Lustre: 372196:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8398.002226] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 8398.005861] Lustre: Skipped 1 previous similar message [ 8398.049909] Lustre: tst250fs-MDT0001: new disk, initializing [ 8398.106838] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8398.145344] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8398.152426] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8400.902123] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 8404.408734] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8409.191524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8409.379936] Lustre: tst250fs-OST0000: new disk, initializing [ 8409.383295] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 8409.387288] Lustre: 373828:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8409.461630] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8414.090665] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 8416.798420] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8416.810818] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8416.891099] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8422.590515] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8422.699554] Lustre: tst250fs-OST0001: new disk, initializing [ 8422.702759] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 8422.708632] Lustre: 374700:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8422.798495] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8422.960090] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8422.970452] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8423.038448] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8427.721927] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_default_debug -1 all [ 8431.940134] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 8448.483305] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8448.490679] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 8448.497065] Lustre: Skipped 2 previous similar messages [ 8448.505380] Lustre: Skipped 3 previous similar messages [ 8453.608683] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 8453.610943] Lustre: Skipped 2 previous similar messages [ 8454.367225] Lustre: server umount tst250fs-MDT0000 complete [ 8456.644909] LustreError: 372189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788369059 with bad export cookie 4857159597821968104 [ 8456.653837] LustreError: MGC192.168.201.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8456.665632] LustreError: 372189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8456.960264] Lustre: server umount tst250fs-MDT0001 complete [ 8468.849517] Lustre: server umount tst250fs-OST0000 complete [ 8481.392410] Lustre: server umount tst250fs-OST0001 complete [ 8493.553650] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 8498.601553] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 8517.330329] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 13:11:59 (1788369119) [ 8518.206923] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 8529.053057] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing set_hostid [ 8535.179855] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing load_modules_local [ 8553.237557] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8263 sec ========== 13:12:35 (1788369155) [ 8554.268660] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 13:12:36 (1788369156) === [ 8556.044729] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 13:12:38 (1788369158) === [ 8579.335237] Lustre: DEBUG MARKER: oleg145-server.virtnet: executing unload_modules_local [ 8581.223970] Key type lgssc unregistered [ 8581.464424] LNet: 383814:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8581.468213] LNetError: 383814:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8581.491491] LNet: Removed LNI 192.168.201.145@tcp [ 8582.076172] Key type .llcrypt unregistered [ 8582.080136] Key type ._llcrypt unregistered