[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 516977388 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003073] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.008819] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010025] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011020] pid_max: default: 32768 minimum: 301 [ 0.012169] LSM: Security Framework initializing [ 0.013069] Yama: becoming mindful. [ 0.014048] SELinux: Initializing. [ 0.015087] *** VALIDATE selinux *** [ 0.024463] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029419] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030189] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031111] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032143] *** VALIDATE tmpfs *** [ 0.033521] *** VALIDATE proc *** [ 0.034335] *** VALIDATE cgroup *** [ 0.035014] *** VALIDATE cgroup2 *** [ 0.037035] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038157] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040037] Spectre V2 : User space: Vulnerable [ 0.041010] Speculative Store Bypass: Vulnerable [ 0.043739] debug: unmapping init [mem 0xffffffffaac59000-0xffffffffaac60fff] [ 0.045192] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046775] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047028] ... version: 2 [ 0.048016] ... bit width: 48 [ 0.049016] ... generic registers: 4 [ 0.050019] ... value mask: 0000ffffffffffff [ 0.051020] ... max period: 00007fffffffffff [ 0.052018] ... fixed-purpose events: 3 [ 0.053012] ... event mask: 000000070000000f [ 0.054286] rcu: Hierarchical SRCU implementation. [ 0.056455] smp: Bringing up secondary CPUs ... [ 0.057609] x86: Booting SMP configuration: [ 0.058032] .... node #0, CPUs: #1 #2 #3 [ 0.062397] smp: Brought up 1 node, 4 CPUs [ 0.064024] smpboot: Max logical packages: 1 [ 0.065020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.128234] node 0 deferred pages initialised in 61ms [ 0.132213] devtmpfs: initialized [ 0.133225] x86/mm: Memory block size: 128MB [ 0.135289] gcov: version magic: 0x41383552 [ 0.136629] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.140079] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.143306] pinctrl core: initialized pinctrl subsystem [ 0.145413] [ 0.146008] ************************************************************* [ 0.150015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.153012] ** ** [ 0.154015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.157018] ** ** [ 0.160019] ** This means that this kernel is built to expose internal ** [ 0.163016] ** IOMMU data structures, which may compromise security on ** [ 0.166016] ** your system. ** [ 0.168014] ** ** [ 0.171017] ** If you see this message and you are not debugging the ** [ 0.174085] ** kernel, report this immediately to your vendor! ** [ 0.177015] ** ** [ 0.179014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.181017] ************************************************************* [ 0.185008] NET: Registered protocol family 16 [ 0.187462] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.190068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.193069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.196118] cpuidle: using governor menu [ 0.197852] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.199613] PCI: Using configuration type 1 for base access [ 0.202132] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.211263] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.213039] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.217117] cryptd: max_cpu_qlen set to 1000 [ 0.219292] ACPI: Added _OSI(Module Device) [ 0.221019] ACPI: Added _OSI(Processor Device) [ 0.223017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.225016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.230534] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.236562] ACPI: Interpreter enabled [ 0.238090] ACPI: PM: (supports S0 S3 S4 S5) [ 0.240018] ACPI: Using IOAPIC for interrupt routing [ 0.241123] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.244329] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.253948] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.257046] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.259081] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.262110] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.266682] acpiphp: Slot [2] registered [ 0.268110] acpiphp: Slot [5] registered [ 0.269111] acpiphp: Slot [6] registered [ 0.270092] acpiphp: Slot [7] registered [ 0.271094] acpiphp: Slot [8] registered [ 0.272098] acpiphp: Slot [9] registered [ 0.273084] acpiphp: Slot [10] registered [ 0.274000] acpiphp: Slot [3] registered [ 0.274000] acpiphp: Slot [4] registered [ 0.275101] acpiphp: Slot [11] registered [ 0.276080] acpiphp: Slot [12] registered [ 0.277078] acpiphp: Slot [13] registered [ 0.278078] acpiphp: Slot [14] registered [ 0.279076] acpiphp: Slot [15] registered [ 0.280100] acpiphp: Slot [16] registered [ 0.282158] acpiphp: Slot [17] registered [ 0.283075] acpiphp: Slot [18] registered [ 0.284100] acpiphp: Slot [19] registered [ 0.285075] acpiphp: Slot [20] registered [ 0.286087] acpiphp: Slot [21] registered [ 0.287071] acpiphp: Slot [22] registered [ 0.288109] acpiphp: Slot [23] registered [ 0.289074] acpiphp: Slot [24] registered [ 0.291107] acpiphp: Slot [25] registered [ 0.292081] acpiphp: Slot [26] registered [ 0.293096] acpiphp: Slot [27] registered [ 0.294162] acpiphp: Slot [28] registered [ 0.296113] acpiphp: Slot [29] registered [ 0.298109] acpiphp: Slot [30] registered [ 0.299132] acpiphp: Slot [31] registered [ 0.301136] PCI host bridge to bus 0000:00 [ 0.302023] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.305024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.307024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.310046] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.312036] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.315027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.316166] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.319011] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.322342] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.332875] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.340075] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.342023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.345018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.347018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.350888] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.354939] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.357046] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.360947] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.366015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.380021] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.384000] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.391204] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.421018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.434017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.469097] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.486246] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.497017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.511018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.540017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.555663] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.566016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.576016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.599019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.610598] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.617022] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.624016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.651016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.663000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.671019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.681020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.700017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.715588] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.731018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.744015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.770021] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.791000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.794664] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.797486] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.801546] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.803282] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.809053] iommu: Default domain type: Passthrough [ 0.810485] SCSI subsystem initialized [ 0.811152] ACPI: bus type USB registered [ 0.812117] usbcore: registered new interface driver usbfs [ 0.814106] usbcore: registered new interface driver hub [ 0.816154] usbcore: registered new device driver usb [ 0.819658] pps_core: LinuxPPS API ver. 1 registered [ 0.823014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.828140] PTP clock support registered [ 0.834000] EDAC MC: Ver: 3.0.0 [ 0.836393] PCI: Using ACPI for IRQ routing [ 0.837000] NetLabel: Initializing [ 0.838016] NetLabel: domain hash size = 128 [ 0.841012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.849781] NetLabel: unlabeled traffic allowed by default [ 0.856372] vgaarb: loaded [ 0.860514] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.865013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.874341] clocksource: Switched to clocksource kvm-clock [ 1.143539] VFS: Disk quotas dquot_6.6.0 [ 1.151390] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.157378] *** VALIDATE ramfs *** [ 1.161910] *** VALIDATE hugetlbfs *** [ 1.169461] pnp: PnP ACPI init [ 1.178043] pnp: PnP ACPI: found 6 devices [ 1.231094] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.237685] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.243500] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.257084] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.263226] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.271469] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.279970] NET: Registered protocol family 2 [ 1.282463] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.295606] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.303296] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.313785] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.319892] TCP: Hash tables configured (established 65536 bind 65536) [ 1.325943] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.333865] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.352472] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.367678] NET: Registered protocol family 1 [ 1.383248] RPC: Registered named UNIX socket transport module. [ 1.389649] RPC: Registered udp transport module. [ 1.393362] RPC: Registered tcp transport module. [ 1.399802] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.406566] NET: Registered protocol family 44 [ 1.409955] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.414462] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.419194] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.423407] PCI: CLS 0 bytes, default 64 [ 1.433442] Unpacking initramfs... [ 4.899176] hrtimer: interrupt took 3809383 ns [ 5.699528] debug: unmapping init [mem 0xffff9136fcc54000-0xffff9136fffbffff] [ 5.703764] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 5.706777] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 5.711629] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 7.415924] Initialise system trusted keyrings [ 7.417638] Key type blacklist registered [ 7.419678] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 7.443765] zbud: loaded [ 7.452746] *** VALIDATE nfs *** [ 7.456379] *** VALIDATE nfs4 *** [ 7.460928] pstore: using deflate compression [ 7.480705] Platform Keyring initialized [ 7.827241] NET: Registered protocol family 38 [ 7.833069] Key type asymmetric registered [ 7.836085] Asymmetric key parser 'x509' registered [ 7.839696] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 7.846634] io scheduler mq-deadline registered [ 7.850347] io scheduler kyber registered [ 7.857536] io scheduler bfq registered [ 7.867116] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 7.879523] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 7.887763] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 7.895944] ACPI: Power Button [PWRF] [ 7.913256] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 7.927073] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 7.959044] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 7.981748] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 8.039487] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 8.088881] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 8.124524] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 8.132744] Non-volatile memory driver v1.3 [ 8.136045] Linux agpgart interface v0.103 [ 8.250728] virtio_blk virtio1: [vda] 149760 512-byte logical blocks (76.7 MB/73.1 MiB) [ 8.258779] vda: detected capacity change from 0 to 76677120 [ 8.318118] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 8.330538] vdb: detected capacity change from 0 to 1073741824 [ 8.365045] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 8.375402] vdc: detected capacity change from 0 to 2621440000 [ 8.423794] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 8.431960] vdd: detected capacity change from 0 to 2621440000 [ 8.463865] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 8.473641] vde: detected capacity change from 0 to 4294967296 [ 8.501806] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 8.511691] vdf: detected capacity change from 0 to 4294967296 [ 8.538978] libphy: Fixed MDIO Bus: probed [ 8.567726] usbcore: registered new interface driver usbserial_generic [ 8.574798] usbserial: USB Serial support registered for generic [ 8.581377] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 8.604260] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 8.616542] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 8.636958] mousedev: PS/2 mouse device common for all mice [ 8.657966] rtc_cmos 00:05: RTC can wake from S4 [ 8.659891] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 8.663082] rtc_cmos 00:05: registered as rtc0 [ 8.675959] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 8.678163] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 8.681066] intel_pstate: CPU model not supported [ 8.691704] hid: raw HID events driver (C) Jiri Kosina [ 8.701286] usbcore: registered new interface driver usbhid [ 8.701531] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 8.708931] usbhid: USB HID core driver [ 8.709545] drop_monitor: Initializing network drop monitor service [ 8.711144] Initializing XFRM netlink socket [ 8.730458] NET: Registered protocol family 10 [ 8.738762] Segment Routing with IPv6 [ 8.741012] NET: Registered protocol family 17 [ 8.744277] mpls_gso: MPLS GSO support [ 8.756908] RAS: Correctable Errors collector initialized. [ 8.761742] AVX version of gcm_enc/dec engaged. [ 8.766626] AES CTR mode by8 optimization enabled [ 8.873748] sched_clock: Marking stable (8873728471, 0)->(9919750041, -1046021570) [ 8.878950] registered taskstats version 1 [ 8.882262] Loading compiled-in X.509 certificates [ 8.886279] zswap: loaded using pool lzo/zbud [ 8.923667] Key type big_key registered [ 8.948400] Key type encrypted registered [ 8.951554] ima: No TPM chip found, activating TPM-bypass! [ 8.962666] ima: Allocated hash algorithm: sha1 [ 8.967881] ima: No architecture policies found [ 8.974029] evm: Initialising EVM extended attributes: [ 8.977883] evm: security.selinux [ 8.985071] evm: security.ima [ 8.987314] evm: security.capability [ 8.989549] evm: HMAC attrs: 0x1 [ 8.996576] rtc_cmos 00:05: setting system clock to 2026-09-03 11:57:38 UTC (1788436658) [ 9.015799] debug: unmapping init [mem 0xffffffffabc03000-0xffffffffabdfffff] [ 9.022286] debug: unmapping init [mem 0xffffffffaa982000-0xffffffffaac58fff] [ 9.034204] Write protecting the kernel read-only data: 28672k [ 9.044349] debug: unmapping init [mem 0xffffffffa9003000-0xffffffffa91fffff] [ 9.049804] debug: unmapping init [mem 0xffffffffa9914000-0xffffffffa99fffff] [ 9.185138] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 9.208842] systemd[1]: Detected virtualization kvm. [ 9.215402] systemd[1]: Detected architecture x86-64. [ 9.218803] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 9.317848] systemd[1]: No hostname configured. [ 9.320889] systemd[1]: Set hostname to . [ 9.327526] random: systemd: uninitialized urandom read (16 bytes read) [ 9.331799] systemd[1]: Initializing machine ID from random generator. [ 9.528108] random: ln: uninitialized urandom read (6 bytes read) [ 9.887678] random: systemd: uninitialized urandom read (16 bytes read) [ 9.899252] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 9.926510] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 9.952042] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Reached target Swap. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Paths. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 11.676640] device-mapper: uevent: version 1.0.3 [ 11.682257] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 13.865913] virtio_net virtio0 ens2: renamed from eth0 Starting dracut initqueue hook... [ 13.947035] random: fast init done [ 14.216793] scsi host0: ata_piix [ 14.231260] scsi host1: ata_piix [ 14.243960] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 14.254236] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 19.504964] random: crng init done [ 19.506456] random: 7 urandom warning(s) missed due to ratelimiting [ 21.410279] dracut-initqueue[593]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 22.907632] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 26.590636] printk: systemd: 26 output lines suppressed due to ratelimiting [ 27.443307] SELinux: Disabled at runtime. [ 27.627748] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 27.652369] systemd[1]: Detected virtualization kvm. [ 27.656263] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 29.789416] systemd[1]: initrd-switch-root.service: Succeeded. [ 29.801805] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 29.826657] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 29.833045] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 29.839092] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 29.852273] systemd[1]: Starting Journal Service... Starting Journal Service... [ 29.899229] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Control Socket. Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Initrd File Systems. Mounting Huge Pages File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Slices. [ 30.271919] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd Root File System. Starting udev Coldplug all Devices... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. Mounting Kernel Debug File System... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 31.483876] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 32.451242] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 32.514113] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 33.025431] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 33.060697] EDAC sbridge: Ver: 1.1.2 [ 36.105642] Key type dns_resolver registered [ 36.640632] NFS: Registering the id_resolver key type [ 36.642702] Key type id_resolver registered [ 36.644328] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started irqbalance daemon. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg424-server login: [ 111.061557] libcfs: loading out-of-tree module taints kernel. [ 111.120446] Key type ._llcrypt registered [ 111.124802] Key type .llcrypt registered [ 111.245821] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [ 130.558342] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 132.195873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 132.217338] alg: No test for adler32 (adler32-zlib) [ 133.802099] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 134.899528] LNet: Added LNI 192.168.204.124@tcp [8/256/0/180] [ 136.800365] Key type lgssc registered [ 138.871717] Lustre: Echo OBD driver; http://www.lustre.org/ [ 162.291769] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 211.546452] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 227.911812] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 227.955537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 229.367448] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 229.436995] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 229.621692] Lustre: lustre-MDT0000: new disk, initializing [ 229.750940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 229.770272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 234.953943] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 250.836395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 250.991658] Lustre: 6535:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 251.075370] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 251.097197] Lustre: Skipped 1 previous similar message [ 251.235428] Lustre: lustre-MDT0001: new disk, initializing [ 251.333306] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 251.369926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 251.387223] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 256.537717] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 262.044648] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 272.774979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 273.077684] Lustre: lustre-OST0000: new disk, initializing [ 273.084830] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 273.091391] Lustre: 8473:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 273.210958] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 279.789967] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 280.608138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 280.631322] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 280.707496] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 293.805211] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 293.975289] Lustre: lustre-OST0001: new disk, initializing [ 293.978416] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 293.982414] Lustre: 9544:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 294.070465] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 300.121633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 300.131212] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 300.219927] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 300.755126] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 314.561967] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 323.046159] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 331.451591] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing check_logdir /tmp/testlogs/ [ 338.025945] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing yml_node [ 342.857068] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 345.996676] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 348.969611] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 351.204693] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 08:03:18 EDT 2026 [ 369.432661] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 371.017366] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 387.553772] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 387.558484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 387.569196] Lustre: Skipped 3 previous similar messages [ 387.574961] Lustre: Skipped 3 previous similar messages [ 392.489831] Lustre: server umount lustre-MDT0000 complete [ 396.110279] LustreError: 8472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788437045 with bad export cookie 17390401295451752910 [ 396.111906] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 396.130317] LustreError: 8472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 397.793233] LustreError: 6540:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 397.793944] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 397.799802] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 397.821718] LustreError: 6540:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 397.846846] Lustre: Skipped 1 previous similar message [ 401.826947] LustreError: 12034:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 401.830399] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 401.848869] LustreError: 12034:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 401.855284] Lustre: Skipped 2 previous similar messages [ 402.475635] Lustre: server umount lustre-MDT0001 complete [ 406.438794] Lustre: server umount lustre-OST0000 complete [ 410.272491] Lustre: server umount lustre-OST0001 complete [ 417.467873] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [ 428.343674] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 473.629305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 474.063122] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 474.112797] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 474.172963] Lustre: lustre-MDT0000: new disk, initializing [ 474.342224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 474.361213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 479.299713] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 491.552313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 491.678112] Lustre: 15115:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 491.729125] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 491.731237] Lustre: Skipped 1 previous similar message [ 491.802299] Lustre: lustre-MDT0001: new disk, initializing [ 491.925300] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 491.967449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 491.998296] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 497.243722] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 507.340780] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 512.794622] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 520.159547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 520.524392] Lustre: lustre-OST0000: new disk, initializing [ 520.531312] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 520.541404] Lustre: 17082:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 520.634077] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 522.100270] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 522.109613] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 522.212371] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 526.474214] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 535.626979] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 540.991784] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 541.335721] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 546.642523] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 546.926821] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 552.929898] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 552.933057] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 552.942155] LustreError: Skipped 1 previous similar message [ 552.950838] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 552.966531] Lustre: Skipped 1 previous similar message [ 552.985503] Lustre: Skipped 1 previous similar message [ 555.429902] Lustre: server umount lustre-OST0000 complete [ 563.171504] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 563.207934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 563.218026] Lustre: Skipped 1 previous similar message [ 565.428697] Lustre: server umount lustre-MDT0000 complete [ 568.297770] LustreError: 15126:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 568.317117] LustreError: 15126:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 569.236045] LustreError: 15107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788437218 with bad export cookie 17390401295451754443 [ 569.244111] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 569.804254] Lustre: server umount lustre-MDT0001 complete [ 575.785407] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 08:07:03 (1788437223) [ 594.552126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 594.937490] LustreError: 19958:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 595.004051] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 599.604810] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 600.034823] LustreError: 19959:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 609.810831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 614.720423] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 622.021695] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 628.885426] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 631.586363] Lustre: Failing over lustre-MDT0000 [ 631.831653] Lustre: server umount lustre-MDT0000 complete [ 635.699797] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 635.717752] Lustre: Skipped 1 previous similar message [ 635.719345] LustreError: 19959:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 635.743419] LustreError: 19959:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 651.462730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 651.591809] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 651.830855] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 655.909374] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 655.914345] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 655.927912] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 656.598936] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 666.639669] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 671.203742] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 671.208572] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 671.228879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 676.793634] Lustre: server umount lustre-MDT0000 complete [ 677.347512] LustreError: 19955:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 677.393578] LustreError: 19955:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 680.913599] LustreError: 19940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788437330 with bad export cookie 17390401295451755857 [ 680.914103] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 680.930516] LustreError: 19940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 681.276534] Lustre: server umount lustre-MDT0001 complete [ 689.567723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 690.027754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 690.035367] Lustre: Skipped 2 previous similar messages [ 694.637990] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 703.857267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 709.206863] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 716.019894] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 721.800420] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 724.073687] Lustre: Failing over lustre-MDT0000 [ 724.307512] Lustre: server umount lustre-MDT0000 complete [ 724.962729] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 724.964819] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 724.981262] Lustre: Skipped 1 previous similar message [ 724.982740] LustreError: 23296:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 725.001047] LustreError: 23296:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 744.842790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 745.149187] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 745.705193] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 748.648638] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 748.650242] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 748.655261] Lustre: Skipped 1 previous similar message [ 748.665515] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 750.263865] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 761.413987] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 766.434650] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 766.449250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 766.455051] Lustre: Skipped 2 previous similar messages [ 766.466625] Lustre: Skipped 3 previous similar messages [ 780.256328] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 780.499469] Lustre: server umount lustre-MDT0000 complete [ 784.403639] LustreError: 25959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788437433 with bad export cookie 17390401295451756795 [ 784.408273] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 784.428254] LustreError: 25959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 784.789818] Lustre: server umount lustre-MDT0001 complete [ 793.842308] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 08:10:41 (1788437441) [ 810.586987] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [ 821.959416] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 864.423729] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 875.197882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 875.640318] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 875.687727] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 875.786390] Lustre: lustre-MDT0000: new disk, initializing [ 875.899029] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 875.903737] Lustre: Skipped 2 previous similar messages [ 875.924438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 881.884511] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 893.832909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 894.112725] Lustre: 29682:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 894.416207] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 898.889592] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 907.178136] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 912.981715] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 919.442229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 919.771752] Lustre: 31654:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 921.783846] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 921.849637] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 921.981932] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 922.006149] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 926.890353] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 940.038462] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 949.821633] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 968.166418] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 968.173971] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 968.183624] LustreError: Skipped 1 previous similar message [ 968.181891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 968.204616] Lustre: Skipped 1 previous similar message [ 968.232734] Lustre: Skipped 7 previous similar messages [ 974.061358] Lustre: server umount lustre-OST0000 complete [ 984.344671] Lustre: server umount lustre-MDT0000 complete [ 988.478206] LustreError: 29673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788437637 with bad export cookie 17390401295451757201 [ 988.489337] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 988.507560] LustreError: 29673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 989.042834] Lustre: server umount lustre-MDT0001 complete [ 1000.552503] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 1003.332879] Key type lgssc unregistered [ 1003.646688] LNet: 33632:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1003.654919] LNetError: 33632:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1003.675324] LNet: Removed LNI 192.168.204.124@tcp [ 1004.744439] Key type .llcrypt unregistered [ 1004.749512] Key type ._llcrypt unregistered [ 1030.174378] Key type ._llcrypt registered [ 1030.177421] Key type .llcrypt registered [ 1030.292815] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 1031.146061] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1031.198161] alg: No test for adler32 (adler32-zlib) [ 1032.260622] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1032.482753] LNet: Added LNI 192.168.204.124@tcp [8/256/0/180] [ 1034.170132] Key type lgssc registered [ 1035.241701] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1044.422919] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1044.460211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1045.862483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1051.037555] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 1056.227113] LustreError: 35040:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1056.262885] LustreError: 35040:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1059.964121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1060.315688] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1064.370161] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 1071.871892] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1077.391828] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1084.563698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1084.895326] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1090.049070] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 1091.564521] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 1099.832858] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1113.977537] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 08:16:01 (1788437761) [ 1116.116848] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1118.204082] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1124.646919] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 08:16:12 (1788437772) [ 1133.047313] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 08:16:20 (1788437780) [ 1134.169242] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1142.592443] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1142.599827] Lustre: Skipped 2 previous similar messages [ 1144.497671] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 08:16:32 (1788437792) [ 1148.302984] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1148.313243] Lustre: Skipped 3 previous similar messages [ 1165.683857] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1165.695992] Lustre: Skipped 6 previous similar messages [ 1168.589511] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 08:16:56 (1788437816) [ 1170.949524] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1174.243466] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1182.437700] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 08:17:10 (1788437830) [ 1184.217974] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1184.221929] Lustre: Skipped 3 previous similar messages [ 1196.251842] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 08:17:24 (1788437844) [ 1212.908206] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1212.928305] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1212.952254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1213.922117] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1213.922660] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1213.929367] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1215.167511] Lustre: server umount lustre-OST0000 complete [ 1219.044216] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1219.048106] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1219.051730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1219.056314] Lustre: Skipped 1 previous similar message [ 1219.062018] Lustre: Skipped 2 previous similar messages [ 1224.161310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1224.169169] Lustre: Skipped 1 previous similar message [ 1224.673935] Lustre: server umount lustre-MDT0000 complete [ 1227.883167] LustreError: 40072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788437877 with bad export cookie 13535352343898467993 [ 1227.885244] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1227.891705] LustreError: 40072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1228.175338] Lustre: server umount lustre-MDT0001 complete [ 1237.974241] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 1240.869285] Key type lgssc unregistered [ 1241.337077] LNet: 42896:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1241.345632] LNetError: 42896:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1241.367723] LNet: Removed LNI 192.168.204.124@tcp [ 1242.217743] Key type .llcrypt unregistered [ 1242.220802] Key type ._llcrypt unregistered [ 1265.754872] Key type ._llcrypt registered [ 1265.758051] Key type .llcrypt registered [ 1266.029503] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [ 1283.687113] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 1285.191670] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1285.232929] alg: No test for adler32 (adler32-zlib) [ 1286.626682] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1287.073243] LNet: Added LNI 192.168.204.124@tcp [8/256/0/180] [ 1288.876384] Key type lgssc registered [ 1290.555434] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1328.109665] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1328.143936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1329.390972] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1329.433536] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1329.493363] Lustre: lustre-MDT0000: new disk, initializing [ 1329.556001] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1329.590949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1333.929206] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 1343.580804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1343.712053] Lustre: 46048:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1343.809345] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1343.814309] Lustre: Skipped 1 previous similar message [ 1343.935217] Lustre: lustre-MDT0001: new disk, initializing [ 1344.048493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1344.093086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1344.106485] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1348.382113] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 1357.610672] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1363.454642] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1370.231097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1370.576239] Lustre: lustre-OST0000: new disk, initializing [ 1370.582442] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1370.589899] Lustre: 48016:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1370.662205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1377.212608] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 1381.011376] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1381.023112] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1381.190380] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1388.823352] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1395.179590] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1395.532405] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1401.553228] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1401.913926] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1406.945429] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1406.952050] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1406.958732] LustreError: Skipped 1 previous similar message [ 1406.968770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1406.984403] Lustre: Skipped 1 previous similar message [ 1407.007923] Lustre: Skipped 1 previous similar message [ 1410.769798] Lustre: server umount lustre-OST0000 complete [ 1415.652545] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1415.669469] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1415.691466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1417.187652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1417.194136] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1421.398889] Lustre: server umount lustre-MDT0000 complete [ 1425.628902] LustreError: 46038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438075 with bad export cookie 581609565907184150 [ 1425.632182] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1425.637297] LustreError: 46038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1426.012548] Lustre: server umount lustre-MDT0001 complete [ 1435.685040] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 08:21:23 (1788438083) [ 1442.811403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1443.284158] LustreError: 50476:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1443.304879] LustreError: 50476:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1443.382975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1448.424791] LustreError: 50477:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1448.520836] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 1452.515696] LustreError: 50476:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1457.121077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1457.562863] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1462.276263] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 1469.157262] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1474.542875] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1480.575661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1480.955391] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1486.317045] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 1493.858287] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1498.297631] LustreError: 53005:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1499.357660] Lustre: Modifying parameter general.timeout=1 in log params [ 1501.086969] Lustre: Modifying parameter general.timeout=3 in log params [ 1501.091207] Lustre: Skipped 1 previous similar message [ 1503.691780] Lustre: Modifying parameter general.timeout=6 in log params [ 1503.699044] Lustre: Skipped 2 previous similar messages [ 1508.730753] Lustre: Modifying parameter general.timeout=11 in log params [ 1508.737289] Lustre: Skipped 4 previous similar messages [ 1517.290264] Lustre: Modifying parameter general.timeout=21 in log params [ 1517.295250] Lustre: Skipped 9 previous similar messages [ 1534.241989] Lustre: Modifying parameter general.timeout=40 in log params [ 1534.250789] Lustre: Skipped 18 previous similar messages [ 1566.779917] Lustre: Modifying parameter general.timeout=76 in log params [ 1566.785779] Lustre: Skipped 35 previous similar messages [ 1624.115316] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 08:24:31 (1788438271) [ 1626.949164] Lustre: 59946:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1628.798389] Lustre: 60044:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1631.401313] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 in log params [ 1631.410421] Lustre: Skipped 58 previous similar messages [ 1632.322792] Lustre: 60259:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1632.339644] Lustre: 60259:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1661.531706] Lustre: 61809:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1661.543125] Lustre: 61809:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1661.567241] LustreError: 61809:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1665.918796] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 08:25:13 (1788438313) [ 1681.385118] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1681.390203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1681.413993] Lustre: Skipped 2 previous similar messages [ 1681.425649] Lustre: Skipped 3 previous similar messages [ 1684.395436] Lustre: server umount lustre-MDT0000 complete [ 1686.505458] LustreError: 50477:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1686.527551] LustreError: 50477:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1688.177042] LustreError: 52320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438337 with bad export cookie 581609565907185032 [ 1688.192520] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1688.202017] LustreError: 52320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1688.652629] Lustre: server umount lustre-MDT0001 complete [ 1703.410512] Lustre: server umount lustre-OST0000 complete [ 1731.626606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1731.999847] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1732.299588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1737.006764] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 1745.974916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1746.085558] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1746.112466] Lustre: 64342:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1751.495475] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 1760.197990] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1766.603411] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1773.406658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1773.631689] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1773.959867] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1773.967675] Lustre: Skipped 1 previous similar message [ 1780.732699] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 1788.890173] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1790.679639] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1790.691857] Lustre: Skipped 18 previous similar messages [ 1790.748214] Lustre: 66914:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1798.115164] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1798.122353] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1798.135243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1804.301383] Lustre: server umount lustre-OST0000 complete [ 1808.865491] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1808.868525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1808.879124] Lustre: Skipped 2 previous similar messages [ 1808.909190] Lustre: Skipped 4 previous similar messages [ 1815.163269] Lustre: server umount lustre-MDT0000 complete [ 1819.107754] LustreError: 66467:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1819.126321] LustreError: 66467:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1819.304099] LustreError: 64335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438468 with bad export cookie 581609565907188028 [ 1819.313895] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1819.316357] LustreError: 64335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1819.667650] Lustre: server umount lustre-MDT0001 complete [ 1831.452706] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 1834.970097] Key type lgssc unregistered [ 1835.406315] LNet: 68124:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1835.422654] LNetError: 68124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1835.442560] LNet: Removed LNI 192.168.204.124@tcp [ 1836.381769] Key type .llcrypt unregistered [ 1836.388452] Key type ._llcrypt unregistered [ 1862.231255] Key type ._llcrypt registered [ 1862.235206] Key type .llcrypt registered [ 1862.384230] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 1863.496194] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1863.719601] alg: No test for adler32 (adler32-zlib) [ 1864.990368] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1865.312459] LNet: Added LNI 192.168.204.124@tcp [8/256/0/180] [ 1866.984216] Key type lgssc registered [ 1868.094736] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1877.259872] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1877.291517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1878.832408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1884.110912] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 1889.255629] LustreError: 69537:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1889.293563] LustreError: 69537:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1894.375467] LustreError: 69536:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1896.447420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1897.152915] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1903.001654] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 1910.928596] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1917.025553] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1924.119233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1924.849937] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1931.315322] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 1938.427736] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1955.846956] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1965.028581] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1965.031220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1965.050586] Lustre: Skipped 2 previous similar messages [ 1965.060941] Lustre: Skipped 2 previous similar messages [ 1970.148085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1970.153185] Lustre: Skipped 2 previous similar messages [ 1970.375228] Lustre: server umount lustre-MDT0000 complete [ 1973.611125] LustreError: 69516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438623 with bad export cookie 10797922653018763955 [ 1973.623131] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1973.981184] Lustre: server umount lustre-MDT0001 complete [ 1987.445934] Lustre: server umount lustre-OST0000 complete [ 2014.113378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2014.401530] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2014.437558] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2014.739965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2019.110751] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2027.981875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2028.107984] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2028.167369] Lustre: 74430:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2028.742696] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2033.742962] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2040.895287] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2046.450723] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2053.022065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2053.192545] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2053.506187] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2059.033107] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2065.067254] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2066.585823] Lustre: 77007:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2068.439987] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2075.106372] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2075.116788] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2075.127651] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2075.617503] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2078.684806] Lustre: server umount lustre-OST0000 complete [ 2084.836216] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2084.847417] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2084.867951] Lustre: Skipped 1 previous similar message [ 2084.880481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2084.891132] Lustre: Skipped 1 previous similar message [ 2089.204036] Lustre: server umount lustre-MDT0000 complete [ 2090.978469] LustreError: 76670:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2090.987387] LustreError: 76670:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2092.816112] LustreError: 74421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438742 with bad export cookie 10797922653018764795 [ 2092.822820] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2092.829552] LustreError: 74421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 2093.148162] Lustre: server umount lustre-MDT0001 complete [ 2102.773265] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 2105.466861] Key type lgssc unregistered [ 2105.654666] LNet: 78220:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2105.660618] LNetError: 78220:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2105.676257] LNet: Removed LNI 192.168.204.124@tcp [ 2106.538669] Key type .llcrypt unregistered [ 2106.545277] Key type ._llcrypt unregistered [ 2129.398987] Key type ._llcrypt registered [ 2129.401939] Key type .llcrypt registered [ 2129.493211] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 2130.442951] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2130.550082] alg: No test for adler32 (adler32-zlib) [ 2131.573931] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2131.773763] LNet: Added LNI 192.168.204.124@tcp [8/256/0/180] [ 2133.410090] Key type lgssc registered [ 2134.451152] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2141.561396] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2141.574983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2142.996195] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2147.508414] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2153.448931] LustreError: 79626:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2153.470970] LustreError: 79626:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2155.702149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2156.172722] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2160.206985] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2166.834252] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2172.527090] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2179.136027] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2179.608192] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2185.626716] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2192.841847] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2198.698978] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2199.669323] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2200.760197] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2203.812418] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2203.822500] Lustre: Skipped 2 previous similar messages [ 2208.822229] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2208.831051] Lustre: Skipped 4 previous similar messages [ 2217.051796] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2217.062412] Lustre: Skipped 8 previous similar messages [ 2233.556903] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2233.560277] Lustre: Skipped 16 previous similar messages [ 2266.022200] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2266.027404] Lustre: Skipped 30 previous similar messages [ 2305.391336] Lustre: 87733:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2308.511921] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 08:35:55 (1788438955) [ 2310.239861] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2312.758303] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 08:36:00 (1788438960) [ 2318.830155] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2318.843066] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2318.862973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2320.867912] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2320.872115] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2320.892187] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2323.254181] Lustre: server umount lustre-OST0000 complete [ 2330.085659] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2330.096398] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2330.114510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2330.121779] Lustre: Skipped 1 previous similar message [ 2333.952848] Lustre: server umount lustre-MDT0000 complete [ 2336.227649] LustreError: 82176:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2336.251224] LustreError: 82176:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2338.224237] LustreError: 87736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438987 with bad export cookie 15403472758479432170 [ 2338.227370] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2338.237343] LustreError: 87736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2338.836801] Lustre: server umount lustre-MDT0001 complete [ 2350.537897] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 2353.175861] Key type lgssc unregistered [ 2353.521232] LNet: 88982:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2353.529495] LNetError: 88982:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2354.601106] LNet: Removed LNI 192.168.204.124@tcp [ 2355.458266] Key type .llcrypt unregistered [ 2355.462970] Key type ._llcrypt unregistered [ 2360.653787] Key type ._llcrypt registered [ 2360.659395] Key type .llcrypt registered [ 2360.795499] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2365.545219] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules [ 2366.372457] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2366.497483] alg: No test for adler32 (adler32-zlib) [ 2367.614393] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2367.901637] LNet: Added LNI 192.168.204.124@tcp [8/256/0/180] [ 2369.616190] Key type lgssc registered [ 2370.719181] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2372.040369] LustreError: 90181:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2387.588269] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 2393.908939] LustreError: 90181:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2396.079678] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2396.119734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2397.729325] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2402.293217] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2407.912241] LustreError: 91208:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2407.932247] LustreError: 91208:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2411.065155] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 08:37:38 (1788439058) [ 2413.032826] LustreError: 91209:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2415.523952] LustreError: 91240:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2415.532513] LustreError: 91240:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2415.539836] LustreError: 91240:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 2415.856435] Lustre: server umount lustre-MDT0000 complete [ 2429.473552] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 2432.253931] Key type lgssc unregistered [ 2432.728297] LNet: 92729:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2432.746392] LNetError: 92729:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2432.773484] LNet: Removed LNI 192.168.204.124@tcp [ 2433.750237] Key type .llcrypt unregistered [ 2433.753281] Key type ._llcrypt unregistered [ 2454.118961] Key type ._llcrypt registered [ 2454.127135] Key type .llcrypt registered [ 2454.288992] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 2455.356277] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2455.443971] alg: No test for adler32 (adler32-zlib) [ 2456.555460] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2456.854219] LNet: Added LNI 192.168.204.124@tcp [8/256/0/180] [ 2458.624609] Key type lgssc registered [ 2459.471664] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2467.684486] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2467.698719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2469.107425] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2473.514289] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2478.574914] LustreError: 93943:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2478.597481] LustreError: 93943:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2481.896665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2482.579898] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2487.369539] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2496.251642] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2501.817212] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2508.966869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2509.540526] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2515.585898] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2523.570147] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2580.158710] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 08:40:27 (1788439227) [ 2582.235466] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2584.235404] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 08:40:31 (1788439231) [ 2589.665974] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2589.680462] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2589.691132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2593.767071] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2593.767626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2593.818739] Lustre: Skipped 1 previous similar message [ 2595.142142] Lustre: server umount lustre-MDT0000 complete [ 2598.884974] LustreError: 93944:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2598.902989] LustreError: 93944:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2599.895363] LustreError: 93922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439249 with bad export cookie 8144301905343089542 [ 2599.905865] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2599.912813] LustreError: 93922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2600.219338] Lustre: server umount lustre-MDT0001 complete [ 2615.228644] Lustre: server umount lustre-OST0000 complete [ 2627.968817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2628.428347] LustreError: 98240:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2628.677094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2632.906365] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2633.698786] LustreError: 98241:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2637.805354] LustreError: 98240:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2641.231464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2641.861296] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2646.327754] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2654.560176] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2660.693314] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2674.042946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2674.241314] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2674.246326] LustreError: 98230:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2674.285053] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2674.291796] LustreError: 100325:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -98 [ 2674.301418] LustreError: 100325:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2674.305834] LustreError: 100325:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2674.374145] Lustre: server umount lustre-OST0000 complete [ 2674.377783] LustreError: 100325:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2686.208372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2686.458242] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2686.463277] LustreError: 98228:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2686.491940] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2686.498355] LustreError: 100792:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -98 [ 2686.502441] LustreError: 100792:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2686.506047] LustreError: 100792:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2686.581059] Lustre: server umount lustre-OST0000 complete [ 2686.589076] LustreError: 100792:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2699.668686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2699.865786] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2699.876654] Lustre: Found index 0 for lustre-OST0000, updating log [ 2699.892234] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2700.052666] Lustre: lustre-OST0000: new disk, initializing [ 2700.078391] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2700.095406] Lustre: 101328:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2700.530187] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2706.746354] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2710.476433] Lustre: Failing over lustre-OST0000 [ 2710.564758] Lustre: server umount lustre-OST0000 complete [ 2716.642168] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2716.647688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2716.657664] Lustre: Skipped 2 previous similar messages [ 2716.678949] Lustre: Skipped 2 previous similar messages [ 2720.686274] Lustre: server umount lustre-MDT0000 complete [ 2721.761644] LustreError: 99254:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2721.769863] LustreError: 99254:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2723.936310] LustreError: 98222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439373 with bad export cookie 8144301905343092286 [ 2723.936825] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2723.946600] LustreError: 98222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2724.291646] Lustre: server umount lustre-MDT0001 complete [ 2733.690223] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 08:43:00 (1788439380) [ 2754.222890] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 2766.846463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2767.304510] LustreError: 103673:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2767.442868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2773.193197] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2784.325984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2790.643810] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2794.046945] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2802.259155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2803.299073] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2803.307372] Lustre: Skipped 1 previous similar message [ 2811.851570] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2824.037538] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2824.141805] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2824.313715] Lustre: lustre-OST0001: new disk, initializing [ 2824.324479] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2824.349060] Lustre: 105993:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2831.616376] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2833.614701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2833.634616] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2833.825720] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2843.879822] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2852.815738] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2859.511650] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2859.518206] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2859.526594] Lustre: Skipped 1 previous similar message [ 2859.550255] Lustre: Skipped 2 previous similar messages [ 2864.511865] Lustre: server umount lustre-MDT0001 complete [ 2864.628205] LustreError: 103673:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2864.660770] LustreError: 103673:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2877.004029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2877.135469] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2877.142511] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2877.232699] Lustre: 103661:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2877.970677] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2877.986848] Lustre: Skipped 1 previous similar message [ 2879.299062] LustreError: 107573:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2879.314089] LustreError: 107573:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2879.324875] LustreError: 103665:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2879.513230] LustreError: 107581:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.124@tcp: cfg command failed: rc = -17 [ 2879.524709] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2879.524709] [ 2883.078875] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2883.165258] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 2894.856911] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 08:45:42 (1788439542) [ 2916.773429] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 2938.720809] Lustre: 109612:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2964.703103] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2968.978611] Lustre: 110748:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3067.361437] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3067.362336] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3067.369438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3067.369447] Lustre: Skipped 1 previous similar message [ 3067.384105] Lustre: Skipped 2 previous similar messages [ 3072.102590] Lustre: server umount lustre-MDT0000 complete [ 3072.484685] LustreError: 103668:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3072.504877] LustreError: 103668:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 3076.793222] LustreError: 104815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439726 with bad export cookie 8144301905343093280 [ 3076.795341] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3076.817531] LustreError: 104815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3077.266582] Lustre: server umount lustre-MDT0001 complete [ 3092.447987] Lustre: server umount lustre-OST0000 complete [ 3094.241444] Lustre: 93253:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788439727/real 1788439727] req@ffff91364c2a8380 x1875314324391936/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788439743 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3094.273714] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3094.297704] Lustre: Skipped 2 previous similar messages [ 3097.354425] Lustre: server umount lustre-OST0001 complete [ 3104.704602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3117.394799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3128.033139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3140.984848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3165.336520] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 3178.502752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3178.546507] Lustre: lustre-MDT0000: reset Object Index mappings [ 3179.337433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3184.156959] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 3194.272185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3194.308581] Lustre: lustre-MDT0001: reset Object Index mappings [ 3200.546664] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 3204.166523] Lustre: 115852:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3212.364726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3219.719913] LustreError: 116207:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3219.746762] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:199 to 0x280000400:225) [ 3219.759854] LustreError: 116207:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3221.360084] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 3223.875666] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:350 to 0x280000401:386) [ 3233.707813] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3240.459258] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3240.488138] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3244.585236] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 3255.060953] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3259.843389] Lustre: 117731:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3321.047510] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 08:52:48 (1788439968) [ 3347.943712] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3347.947186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3347.958621] Lustre: Skipped 2 previous similar messages [ 3347.975492] Lustre: Skipped 6 previous similar messages [ 3351.310806] Lustre: server umount lustre-MDT0000 complete [ 3355.324753] LustreError: 114682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440004 with bad export cookie 8144301905343140152 [ 3355.332277] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3355.343737] LustreError: 114682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3362.077991] Lustre: server umount lustre-MDT0001 complete [ 3381.216138] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3381.329587] Lustre: server umount lustre-OST0000 complete [ 3391.957235] Lustre: server umount lustre-OST0001 complete [ 3400.007755] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [ 3411.609431] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 3457.802072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3458.119753] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3458.158284] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3458.329721] Lustre: lustre-MDT0000: new disk, initializing [ 3458.489679] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3458.502261] Lustre: Skipped 3 previous similar messages [ 3458.548446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3465.574506] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 3474.694615] Lustre: server umount lustre-MDT0000 complete [ 3487.287539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3487.729920] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3487.767749] Lustre: 122008:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3487.780807] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3493.294737] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 3498.782398] Lustre: server umount lustre-MDT0000 complete [ 3509.587765] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 08:55:57 (1788440157) [ 3511.541675] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3513.737694] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 08:56:01 (1788440161) [ 3534.529193] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 3545.558933] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3551.729942] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 3562.407772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3562.516815] Lustre: 124056:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3562.576558] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3562.581196] Lustre: Skipped 1 previous similar message [ 3562.681603] Lustre: lustre-MDT0001: new disk, initializing [ 3562.824466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3562.832417] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3567.831479] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 3573.244332] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3581.149104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3581.530264] Lustre: lustre-OST0000: new disk, initializing [ 3581.537422] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3581.543115] Lustre: 125640:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3583.077684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3583.095357] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3583.275876] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3589.341723] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 3602.957818] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3603.158527] Lustre: lustre-OST0001: new disk, initializing [ 3603.161878] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3603.167747] Lustre: 126514:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3604.815071] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3604.822395] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3604.987253] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3610.374487] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 3621.764937] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3626.281399] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4084.232956] Lustre: 132952:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4084.241907] Lustre: 132952:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4084.248768] Lustre: 132952:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4084.254669] Lustre: 132952:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4084.262274] Lustre: 132952:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4084.273498] Lustre: 132952:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4426.101620] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 09:11:13 (1788441073) [ 4443.630541] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4443.639149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4443.652318] Lustre: Skipped 6 previous similar messages [ 4443.654161] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4443.660992] Lustre: Skipped 7 previous similar messages [ 4444.611706] Lustre: server umount lustre-MDT0000 complete [ 4448.749657] LustreError: 127338:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4448.784401] LustreError: 127338:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 4449.152316] LustreError: 124051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441098 with bad export cookie 8144301905343210117 [ 4449.155274] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4449.163099] LustreError: 124051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4453.864753] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4453.884781] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4453.893897] Lustre: Skipped 1 previous similar message [ 4455.776206] Lustre: server umount lustre-MDT0001 complete [ 4466.526245] Lustre: server umount lustre-OST0000 complete [ 4477.889617] Lustre: server umount lustre-OST0001 complete [ 4486.567898] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [ 4498.915773] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 4539.901922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4540.216902] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4540.268706] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4540.372754] Lustre: lustre-MDT0000: new disk, initializing [ 4540.439857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4540.449663] Lustre: Skipped 5 previous similar messages [ 4540.463468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4545.214246] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4556.472283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4556.641258] Lustre: 136011:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4556.757634] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4556.772838] Lustre: Skipped 1 previous similar message [ 4556.886901] Lustre: lustre-MDT0001: new disk, initializing [ 4557.042047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4557.049225] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4561.658845] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4575.689066] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4584.366745] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4593.116379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4593.356360] Lustre: lustre-OST0000: new disk, initializing [ 4593.362938] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4593.372333] Lustre: 137981:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4594.869962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4594.875865] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4595.013745] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4598.546813] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 4611.833821] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4625.077310] Lustre: lustre-MDD0000: changelog on [ 4628.409994] Lustre: lustre-MDD0001: changelog on [ 6864.671221] Lustre: lustre-MDD0001: changelog off [ 6867.081625] Lustre: 136019:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6868.117226] Lustre: lustre-MDD0000: changelog off [ 6878.689661] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6878.697953] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6878.705124] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6878.718406] Lustre: Skipped 2 previous similar messages [ 6878.739187] Lustre: Skipped 3 previous similar messages [ 6880.938814] Lustre: server umount lustre-OST0000 complete [ 6886.881362] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6886.896104] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6886.921935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6888.933356] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6888.937119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6888.978504] Lustre: Skipped 1 previous similar message [ 6894.051506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6894.056424] Lustre: Skipped 1 previous similar message [ 6899.168541] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6899.170350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6899.186753] Lustre: Skipped 1 previous similar message [ 6899.477069] Lustre: server umount lustre-MDT0000 complete [ 6904.032520] LustreError: 136004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443553 with bad export cookie 8144301905344036362 [ 6904.041736] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6904.046968] LustreError: 136004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6904.716846] Lustre: server umount lustre-MDT0001 complete [ 6917.287823] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 6920.436866] Key type lgssc unregistered [ 6920.817816] LNet: 147178:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6920.827796] LNetError: 147178:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6920.860458] LNet: Removed LNI 192.168.204.124@tcp [ 6921.955166] Key type .llcrypt unregistered [ 6921.957430] Key type ._llcrypt unregistered [ 6944.431105] Key type ._llcrypt registered [ 6944.438949] Key type .llcrypt registered [ 6944.542782] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [ 6960.837125] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 6962.064430] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6962.086242] alg: No test for adler32 (adler32-zlib) [ 6963.216379] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 6963.590214] LNet: Added LNI 192.168.204.124@tcp [8/256/0/180] [ 6965.313721] Key type lgssc registered [ 6966.598811] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7004.064560] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7004.113515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7005.396666] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7005.436694] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7005.538673] Lustre: lustre-MDT0000: new disk, initializing [ 7005.609155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7005.626418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7010.223356] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 7022.940798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7023.087329] Lustre: 150374:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7023.171608] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7023.179707] Lustre: Skipped 1 previous similar message [ 7023.278511] Lustre: lustre-MDT0001: new disk, initializing [ 7023.453925] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7023.530940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7023.557406] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7028.581213] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 7038.721597] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7046.116993] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7055.584535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7056.010467] Lustre: lustre-OST0000: new disk, initializing [ 7056.018426] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7056.031900] Lustre: 152346:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7056.165054] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7062.646364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7062.665670] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7062.857713] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7064.071038] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 7074.867159] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7080.169518] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_module obdecho/obdecho [ 7082.285271] LustreError: 153349:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 7082.296964] LustreError: 153349:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 7084.239811] LustreError: 153445:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 7098.850191] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7098.857057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7098.872235] Lustre: Skipped 2 previous similar messages [ 7098.885893] Lustre: Skipped 2 previous similar messages [ 7103.397567] Lustre: server umount lustre-MDT0000 complete [ 7107.074059] LustreError: 150366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443756 with bad export cookie 2135355630167315594 [ 7107.075145] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7107.088172] LustreError: 150366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7107.271204] Lustre: server umount lustre-MDT0001 complete [ 7121.498286] Lustre: server umount lustre-OST0000 complete [ 7145.563535] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [ 7160.079486] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 7203.816890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7204.284916] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7204.339498] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7204.448214] Lustre: lustre-MDT0000: new disk, initializing [ 7204.535368] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7204.548441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7210.044204] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 7222.872797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7223.063534] Lustre: 157184:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7223.162043] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7223.168053] Lustre: Skipped 1 previous similar message [ 7223.237952] Lustre: lustre-MDT0001: new disk, initializing [ 7223.322459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7223.394148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7223.408964] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7227.688572] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 7238.959545] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7247.073743] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7256.950119] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7257.278858] Lustre: lustre-OST0000: new disk, initializing [ 7257.286418] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7257.292749] Lustre: 159152:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7257.472130] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7263.793752] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7263.802282] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7263.944440] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7264.576477] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 7275.965513] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7281.994762] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7282.253770] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7287.638923] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7287.858599] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7294.948480] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7294.950722] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7294.952141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7294.967878] Lustre: Skipped 1 previous similar message [ 7296.497591] Lustre: server umount lustre-OST0000 complete [ 7305.185481] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7305.192698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7305.197776] Lustre: Skipped 1 previous similar message [ 7305.210406] Lustre: Skipped 2 previous similar messages [ 7307.342655] Lustre: server umount lustre-MDT0000 complete [ 7310.306832] LustreError: 157192:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7310.337057] LustreError: 157192:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7312.149680] LustreError: 157176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443961 with bad export cookie 2135355630167316553 [ 7312.152623] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7312.156975] LustreError: 157176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7312.545403] Lustre: server umount lustre-MDT0001 complete [ 7317.668617] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 09:59:24 (1788443964) [ 7319.577442] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7321.975322] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 09:59:29 (1788443969) [ 7331.670362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7332.231172] LustreError: 161566:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7332.365239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7337.447959] LustreError: 161567:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7337.834418] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 7341.541225] LustreError: 161566:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7346.666436] LustreError: 161567:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7348.769389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7353.639721] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 7362.622568] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7369.056915] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7376.679532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7377.153752] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7377.161658] Lustre: Skipped 1 previous similar message [ 7383.324152] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 7392.144282] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7404.007405] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7404.011695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7404.027311] Lustre: Skipped 2 previous similar messages [ 7404.035430] Lustre: Skipped 2 previous similar messages [ 7404.123930] Lustre: server umount lustre-MDT0000 complete [ 7407.620169] LustreError: 162306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444057 with bad export cookie 2135355630167317435 [ 7407.620754] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7407.623746] LustreError: 162306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7408.272534] LustreError: 162770:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7408.296638] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.24@tcp (stopping) [ 7409.127564] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7414.087665] Lustre: server umount lustre-MDT0001 complete [ 7422.421779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7441.360160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7441.612387] LustreError: 165420:0:(llog.c:1655:llog_backup()) MGC192.168.204.124@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7441.623288] Lustre: 165420:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.124@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7441.825753] LustreError: 165425:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7441.835582] LustreError: 165425:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7474.823125] LustreError: 165425:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7474.840422] LustreError: 165425:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [ 7502.817479] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7502.834549] Lustre: 165420:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.124@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7540.360582] LustreError: 165425:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7540.388933] LustreError: 165425:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 39 previous similar messages [ 7543.791951] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7548.917467] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7550.721477] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 7560.641827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7577.569749] LustreError: 148597:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000004c01b111 x1875319050082816/t0(0) o250->MGC192.168.204.124@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7577.619907] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7583.215648] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7583.277881] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 7591.531616] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7593.328659] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7599.688868] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7601.604178] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7624.163087] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7624.165103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7624.198395] Lustre: Skipped 1 previous similar message [ 7624.217717] Lustre: Skipped 5 previous similar messages [ 7624.664652] Lustre: server umount lustre-MDT0000 complete [ 7629.447378] LustreError: 161547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444278 with bad export cookie 2135355630167323924 [ 7629.452641] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7629.456494] LustreError: 161547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7629.788259] Lustre: server umount lustre-MDT0001 complete [ 7643.783761] Lustre: server umount lustre-OST0000 complete [ 7668.635298] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [ 7684.069475] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 7732.689905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7732.923268] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7732.958433] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7733.056159] Lustre: lustre-MDT0000: new disk, initializing [ 7733.214538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7733.230235] Lustre: Skipped 1 previous similar message [ 7733.270376] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7738.866832] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 7750.588522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7750.740462] Lustre: 170548:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7751.089317] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7756.620874] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 7768.143816] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7775.635376] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7785.036611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7785.309622] Lustre: lustre-OST0000: new disk, initializing [ 7785.312532] Lustre: Skipped 1 previous similar message [ 7785.315841] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7785.320048] Lustre: Skipped 2 previous similar messages [ 7785.325203] Lustre: 172518:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7791.955815] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 7792.219392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7792.230064] Lustre: Skipped 1 previous similar message [ 7792.238869] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7792.378759] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7804.328755] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7809.766547] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7810.014325] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7815.281519] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7815.551689] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7818.219783] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7818.225564] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7818.234082] LustreError: Skipped 1 previous similar message [ 7818.254830] Lustre: Skipped 2 previous similar messages [ 7818.258047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7818.264048] Lustre: Skipped 1 previous similar message [ 7823.844965] Lustre: server umount lustre-OST0000 complete [ 7827.938425] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7842.272142] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7842.467591] Lustre: server umount lustre-MDT0000 complete [ 7843.816080] LustreError: 170560:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7843.836499] LustreError: 170560:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 35 previous similar messages [ 7847.477641] LustreError: 170543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444496 with bad export cookie 2135355630167340731 [ 7847.486471] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7847.489369] LustreError: 170543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7848.076170] Lustre: server umount lustre-MDT0001 complete [ 7858.770615] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 10:08:26 (1788444506) [ 7865.646995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7871.595402] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 7881.823639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7887.863580] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 7897.522642] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7904.750769] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7913.076909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7921.175888] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 7929.979939] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7953.423804] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 10:10:00 (1788444600) [ 7962.081071] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7962.084718] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7962.093728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7962.103472] Lustre: Skipped 3 previous similar messages [ 7962.120210] Lustre: Skipped 10 previous similar messages [ 7964.570462] Lustre: server umount lustre-OST0000 complete [ 7975.081879] Lustre: server umount lustre-MDT0000 complete [ 7979.392188] LustreError: 174914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444628 with bad export cookie 2135355630167341613 [ 7979.406552] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7979.425639] LustreError: 174914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7979.831495] Lustre: server umount lustre-MDT0001 complete [ 7992.652909] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 7995.977468] Key type lgssc unregistered [ 7996.342237] LNet: 178685:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7996.348873] LNetError: 178685:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7996.363560] LNet: Removed LNI 192.168.204.124@tcp [ 7997.423175] Key type .llcrypt unregistered [ 7997.424948] Key type ._llcrypt unregistered [ 8021.485197] Key type ._llcrypt registered [ 8021.487878] Key type .llcrypt registered [ 8021.692465] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 8023.029544] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8023.160076] alg: No test for adler32 (adler32-zlib) [ 8024.419488] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8024.596771] LNet: Added LNI 192.168.204.124@tcp [8/256/0/180] [ 8026.216155] Key type lgssc registered [ 8027.473696] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8038.527635] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8038.541414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8054.049089] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8059.168358] LustreError: 179926:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.124@tcp: failed processing log, type 4: rc = -110 [ 8064.288483] LustreError: 179926:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8064.323062] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 8064.342901] LustreError: MGC192.168.204.124@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 8064.372293] LustreError: 179926:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 8064.398220] LustreError: 179926:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -22 [ 8064.409950] LustreError: 179926:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 8064.492960] Lustre: server umount lustre-OST0000 complete [ 8064.495344] LustreError: 179926:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8073.704606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8075.209164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8079.603149] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 8085.488500] LustreError: 180426:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8085.508168] LustreError: 180426:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8088.971806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8089.387086] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8094.750485] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 8103.259911] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8110.191956] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8118.135090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8118.417699] LustreError: 182243:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8118.426174] Lustre: 182243:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.124@tcp: invalid local config log lustre-OST0000: rc = -22 [ 8118.642646] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8126.447188] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 8135.816549] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8139.235940] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8139.242071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8139.266499] Lustre: Skipped 1 previous similar message [ 8139.286366] Lustre: Skipped 2 previous similar messages [ 8144.356611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8144.864605] Lustre: server umount lustre-MDT0000 complete [ 8148.996744] LustreError: 180404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444798 with bad export cookie 12715519662358450848 [ 8149.015946] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8149.348986] Lustre: server umount lustre-MDT0001 complete [ 8163.975181] Lustre: server umount lustre-OST0000 complete [ 8175.805412] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [ 8189.723918] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 8233.866527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8234.264995] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8234.307184] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8234.415877] Lustre: lustre-MDT0000: new disk, initializing [ 8234.566364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8234.584291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8239.562229] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 8252.666203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8252.825877] Lustre: 185790:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8252.892272] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8252.897523] Lustre: Skipped 1 previous similar message [ 8252.988478] Lustre: lustre-MDT0001: new disk, initializing [ 8253.121970] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8253.215767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8253.240182] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8260.095075] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 8272.360948] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8278.450304] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8287.995454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8288.391715] Lustre: lustre-OST0000: new disk, initializing [ 8288.400222] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8288.413712] Lustre: 187759:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8288.606050] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8294.486267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8294.502380] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8294.670969] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8296.739587] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 8308.187306] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8314.400125] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8314.637516] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8321.953207] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8322.398459] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8325.603313] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8325.628939] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8325.663870] Lustre: Skipped 1 previous similar message [ 8325.675470] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8325.685496] Lustre: Skipped 3 previous similar messages [ 8330.726312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8330.737174] Lustre: Skipped 1 previous similar message [ 8331.151079] Lustre: server umount lustre-OST0000 complete [ 8335.843293] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8335.858209] Lustre: Skipped 1 previous similar message [ 8335.864475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8340.458738] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8340.489401] LustreError: Skipped 1 previous similar message [ 8340.494671] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8341.877648] Lustre: server umount lustre-MDT0000 complete [ 8346.081925] LustreError: 185802:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8346.105197] LustreError: 185802:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8346.820276] LustreError: 185784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444996 with bad export cookie 12715519662358451569 [ 8346.828560] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8346.834075] LustreError: 185784:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 8347.141641] Lustre: server umount lustre-MDT0001 complete [ 8357.695993] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 10:16:45 (1788445005) [ 8381.779662] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 8386.053963] Key type lgssc unregistered [ 8386.490636] LNet: 190704:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8386.504438] LNetError: 190704:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8387.576103] LNet: Removed LNI 192.168.204.124@tcp [ 8388.646897] Key type .llcrypt unregistered [ 8388.648704] Key type ._llcrypt unregistered [ 8411.504792] Key type ._llcrypt registered [ 8411.506512] Key type .llcrypt registered [ 8411.660744] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 8413.037478] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8413.112623] alg: No test for adler32 (adler32-zlib) [ 8414.228511] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8414.592898] LNet: Added LNI 192.168.204.124@tcp [8/256/0/180] [ 8416.368186] Key type lgssc registered [ 8417.957760] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8427.535194] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8427.561935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8429.079991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8434.377783] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 8439.274982] LustreError: 191921:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8439.299116] LustreError: 191921:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8444.394642] LustreError: 191922:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8444.833771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8445.475021] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8451.638570] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 8460.469311] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8467.961917] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8476.877053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8477.160714] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8477.168893] LustreError: 193787:0:(llog.c:1696:llog_backup()) MGC192.168.204.124@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8477.183442] Lustre: 193787:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.124@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8477.324894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8484.695099] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 8494.110604] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8496.609734] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8496.620880] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8496.642422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8500.194261] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8500.195901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8500.228023] Lustre: Skipped 1 previous similar message [ 8500.251653] Lustre: Skipped 2 previous similar messages [ 8502.820587] Lustre: server umount lustre-MDT0000 complete [ 8505.324494] LustreError: 192848:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8505.354491] LustreError: 192848:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8507.304672] LustreError: 191902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445156 with bad export cookie 8738092832623490380 [ 8507.307734] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8507.317151] LustreError: 191902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8507.809168] Lustre: server umount lustre-MDT0001 complete [ 8521.674221] Lustre: server umount lustre-OST0000 complete [ 8532.071598] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [ 8543.490125] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 8588.065816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8588.291942] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8588.331436] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8588.431401] Lustre: lustre-MDT0000: new disk, initializing [ 8588.517547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8588.535834] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8593.486639] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 8606.239932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8606.344848] Lustre: 197319:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8606.383712] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8606.387717] Lustre: Skipped 1 previous similar message [ 8606.390892] LustreError: 198103:0:(llog.c:1696:llog_backup()) MGC192.168.204.124@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8606.397941] LustreError: 198103:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8606.402405] Lustre: 198103:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.124@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8606.419432] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8606.423530] Lustre: Skipped 1 previous similar message [ 8606.498497] Lustre: lustre-MDT0001: new disk, initializing [ 8606.596856] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8606.631178] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8606.661502] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8611.792561] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 8622.753500] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8629.296972] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8637.081660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8637.337964] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8637.342179] Lustre: Skipped 1 previous similar message [ 8637.345802] LustreError: 199265:0:(llog.c:1696:llog_backup()) MGC192.168.204.124@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8637.353845] LustreError: 199265:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8637.358848] Lustre: 199265:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.124@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8637.368925] Lustre: 199265:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8637.381791] Lustre: lustre-OST0000: new disk, initializing [ 8637.385105] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8637.389954] Lustre: 199289:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8637.448420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8643.510940] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 8646.186888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8646.194077] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8646.343053] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8653.103420] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8658.762645] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8659.027465] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8664.707166] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8664.998132] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8672.225294] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8672.232858] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8672.235425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8672.239936] Lustre: Skipped 1 previous similar message [ 8672.247301] Lustre: Skipped 1 previous similar message [ 8673.388119] Lustre: server umount lustre-OST0000 complete [ 8677.346544] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8677.354991] Lustre: Skipped 1 previous similar message [ 8677.359703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8677.363889] Lustre: Skipped 1 previous similar message [ 8682.466697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8682.477882] Lustre: Skipped 1 previous similar message [ 8683.558715] LustreError: 200830:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000f4d255f2) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 8685.664481] Lustre: server umount lustre-MDT0000 complete [ 8687.587893] LustreError: 197330:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8687.620826] LustreError: 197330:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8691.046603] LustreError: 197311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445340 with bad export cookie 8738092832623491094 [ 8691.047979] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8691.055329] LustreError: 197311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8691.510703] Lustre: server umount lustre-MDT0001 complete [ 8703.286669] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 10:22:30 (1788445350) [ 8724.316414] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 8736.414285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8736.971928] LustreError: 202418:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8737.062420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8744.101429] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 8747.496773] LustreError: 202419:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8747.529245] LustreError: 202419:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8756.503091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8761.289533] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 8764.796189] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8772.986214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8773.645343] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8773.652558] Lustre: Skipped 1 previous similar message [ 8780.555287] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 8790.442061] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8790.817537] Lustre: lustre-OST0001: new disk, initializing [ 8790.827705] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8790.838747] Lustre: 204726:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8791.441291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8791.466528] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8791.664285] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8797.530194] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 8813.256690] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8821.584474] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8824.380109] Lustre: DEBUG MARKER: ADD OST3 [ 8830.535517] loop: module loaded [ 8830.601458] loop0: detected capacity change from 0 to 204800000 [ 8830.733802] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8830.739818] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8830.745873] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8837.457237] loop0: detected capacity change from 0 to 204800000 [ 8838.078452] Lustre: DEBUG MARKER: STOP OST3 [ 8844.165753] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8844.403817] Lustre: lustre-OST0003: new disk, initializing [ 8844.409615] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8844.420833] Lustre: 206743:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8844.593587] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8844.598037] Lustre: Skipped 1 previous similar message [ 8846.589902] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8846.719428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8846.728279] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8846.788878] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8848.503300] Lustre: Failing over lustre-OST0003 [ 8848.619450] LustreError: 206907:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8848.674516] Lustre: server umount lustre-OST0003 complete [ 8851.638672] LustreError: 204336:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8851.651271] LustreError: 204336:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8851.953067] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8851.958845] LustreError: Skipped 1 previous similar message [ 8851.969478] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8851.984259] Lustre: Skipped 1 previous similar message [ 8852.926375] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 8861.937675] Lustre: DEBUG MARKER: START OST3 again [ 8869.375255] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8869.834392] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8871.021360] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8871.829081] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8871.832516] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8877.435762] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 8901.579782] Lustre: Failing over lustre-OST0003 [ 8901.705088] Lustre: server umount lustre-OST0003 complete [ 8902.625192] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8902.627267] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8902.631386] LustreError: Skipped 1 previous similar message [ 8902.634337] LustreError: 203903:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8902.645682] Lustre: Skipped 1 previous similar message [ 8902.681605] LustreError: 203903:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 8921.057739] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8921.072033] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8921.076192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8921.082522] Lustre: Skipped 1 previous similar message [ 8935.392155] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8935.635244] Lustre: server umount lustre-MDT0000 complete [ 8939.632067] LustreError: 202399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445589 with bad export cookie 8738092832623491976 [ 8939.641636] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8939.651583] LustreError: 202399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8941.551608] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8941.559584] Lustre: Skipped 12 previous similar messages [ 8946.087263] Lustre: server umount lustre-MDT0001 complete [ 8950.699289] Lustre: server umount lustre-OST0000 complete [ 8955.337396] Lustre: server umount lustre-OST0001 complete [ 8964.489574] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [ 8974.762056] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 9023.890943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9024.081592] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9024.116530] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9024.207503] Lustre: lustre-MDT0000: new disk, initializing [ 9024.344786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9024.352594] Lustre: Skipped 1 previous similar message [ 9024.385276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9030.227760] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 9043.354504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9043.521211] Lustre: 212192:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9043.581511] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9043.588513] Lustre: Skipped 1 previous similar message [ 9043.659939] Lustre: lustre-MDT0001: new disk, initializing [ 9043.810419] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9043.823964] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9049.269567] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 9061.070550] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9068.318880] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9076.505833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9076.911266] Lustre: lustre-OST0000: new disk, initializing [ 9076.917312] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9076.931324] Lustre: 214163:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9078.345480] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9078.375384] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9078.487085] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9085.226292] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 9097.233952] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9103.273627] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9103.530630] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9109.313960] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9109.666660] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9114.595336] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9114.596213] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9114.613501] LustreError: Skipped 1 previous similar message [ 9114.615670] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9114.615679] Lustre: Skipped 2 previous similar messages [ 9114.662381] Lustre: Skipped 5 previous similar messages [ 9126.368131] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9126.584071] Lustre: server umount lustre-OST0000 complete [ 9145.317047] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9145.595629] Lustre: server umount lustre-MDT0000 complete [ 9149.798895] LustreError: 212989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445799 with bad export cookie 8738092832623494265 [ 9149.808968] LustreError: 212989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9149.815756] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9150.131247] Lustre: server umount lustre-MDT0001 complete [ 9154.449762] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 10:30:02 (1788445802) [ 9172.999761] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [ 9187.074114] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 9230.896553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9231.095986] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9231.140264] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9231.219814] Lustre: lustre-MDT0000: new disk, initializing [ 9231.313426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9236.071521] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 9246.653045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9246.796942] Lustre: 218780:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9247.085428] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9252.621123] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 9262.641583] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9270.957409] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9280.251405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9280.680246] Lustre: 220752:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9280.751976] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9280.759331] Lustre: Skipped 4 previous similar messages [ 9281.984511] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9282.219712] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9289.465890] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 9301.332408] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9306.781308] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9306.983049] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9311.776919] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9312.022873] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9317.860080] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9317.866065] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9317.870823] LustreError: Skipped 1 previous similar message [ 9317.886600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9317.896316] Lustre: Skipped 3 previous similar messages [ 9317.913602] Lustre: Skipped 12 previous similar messages [ 9320.369108] Lustre: server umount lustre-OST0000 complete [ 9330.473837] Lustre: server umount lustre-MDT0000 complete [ 9333.217713] LustreError: 218792:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9333.244134] LustreError: 218792:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 19 previous similar messages [ 9334.578714] LustreError: 220750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445984 with bad export cookie 8738092832623495154 [ 9334.592615] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9334.599111] LustreError: 220750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9335.050315] Lustre: server umount lustre-MDT0001 complete [ 9343.595389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9348.831206] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 9357.991493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9363.168633] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 9370.936893] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9377.504671] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9384.652472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9391.105648] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 9399.316787] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9413.611235] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9414.949921] Lustre: server umount lustre-OST0000 complete [ 9427.348632] Lustre: server umount lustre-MDT0000 complete [ 9431.675904] LustreError: 222951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446081 with bad export cookie 8738092832623496036 [ 9431.689825] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9431.694462] LustreError: 222951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9432.086727] Lustre: server umount lustre-MDT0001 complete [ 9540.583924] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [ 9543.640735] Key type lgssc unregistered [ 9544.073592] LNet: 226635:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9544.088196] LNetError: 226635:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9544.109708] LNet: Removed LNI 192.168.204.124@tcp [ 9545.209294] Key type .llcrypt unregistered [ 9545.214741] Key type ._llcrypt unregistered [ 9571.091645] Key type ._llcrypt registered [ 9571.109676] Key type .llcrypt registered [ 9571.244845] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [ 9587.893855] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 9588.965532] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9588.990236] alg: No test for adler32 (adler32-zlib) [ 9590.246882] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 9590.516331] LNet: Added LNI 192.168.204.124@tcp [8/256/0/180] [ 9592.232415] Key type lgssc registered [ 9593.622435] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9632.035214] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9632.082677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9633.390635] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9633.437535] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9633.542794] Lustre: lustre-MDT0000: new disk, initializing [ 9633.613414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9633.633107] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9638.755423] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 9650.459547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9650.685075] Lustre: 229990:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9650.755694] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9650.776601] Lustre: Skipped 1 previous similar message [ 9650.917693] Lustre: lustre-MDT0001: new disk, initializing [ 9651.107853] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9651.167654] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9651.185246] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9656.178789] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 9665.873085] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9671.609434] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9678.644243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9678.928361] Lustre: lustre-OST0000: new disk, initializing [ 9678.933364] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9678.940209] Lustre: 231958:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9679.016228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9685.115842] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 9686.096607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9686.109512] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9686.261317] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9694.662319] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9699.474615] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9699.636369] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9704.266732] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9704.497545] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9706.987466] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9706.991996] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9706.997932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9707.011510] Lustre: Skipped 1 previous similar message [ 9707.044092] LustreError: Skipped 1 previous similar message [ 9712.101393] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9712.112123] Lustre: Skipped 2 previous similar messages [ 9712.567961] Lustre: server umount lustre-OST0000 complete [ 9717.226889] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9717.237414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9717.245047] Lustre: Skipped 1 previous similar message [ 9722.342994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9722.348668] Lustre: Skipped 1 previous similar message [ 9722.912950] Lustre: server umount lustre-MDT0000 complete [ 9726.592032] LustreError: 229981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446376 with bad export cookie 13953182370706371182 [ 9726.600668] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9726.608678] LustreError: 229981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9726.929866] Lustre: server umount lustre-MDT0001 complete [ 9748.011568] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 9758.246429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9758.668175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9763.492560] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 9763.815466] LustreError: 234890:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9763.842860] LustreError: 234890:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9768.933176] LustreError: 234891:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9773.185268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9773.628319] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9778.081464] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 9780.905732] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9787.234671] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9787.587320] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9793.801198] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 9802.119198] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9802.380844] Lustre: lustre-OST0001: new disk, initializing [ 9802.386598] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9802.402433] Lustre: 237199:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9803.146250] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9803.152869] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9803.281468] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9808.426342] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 9820.198460] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9831.799362] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9844.830420] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 10:41:32 (1788446492) [ 9848.290754] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9848.295233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9848.335659] Lustre: Skipped 3 previous similar messages [ 9848.341610] Lustre: Skipped 1 previous similar message [ 9858.535267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9858.539198] Lustre: Skipped 7 previous similar messages [ 9861.088545] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9861.337799] Lustre: server umount lustre-MDT0000 complete [ 9863.652254] LustreError: 234890:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9863.684930] LustreError: 234890:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9865.166732] LustreError: 237967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446514 with bad export cookie 13953182370706372057 [ 9865.177913] LustreError: 237967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9865.180087] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9865.473177] Lustre: server umount lustre-MDT0001 complete [ 9879.600045] Lustre: server umount lustre-OST0000 complete [ 9894.141360] Lustre: server umount lustre-OST0001 complete [ 9900.958616] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [ 9912.079098] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [ 9949.592991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9949.805655] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9949.836471] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9949.924926] Lustre: lustre-MDT0000: new disk, initializing [ 9950.046783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9950.054550] Lustre: Skipped 1 previous similar message [ 9950.074411] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9954.411414] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 9963.687249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9963.762505] Lustre: 241148:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9963.819757] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9963.825091] Lustre: Skipped 1 previous similar message [ 9963.886850] Lustre: lustre-MDT0001: new disk, initializing [ 9963.989474] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9963.996591] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9967.748468] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [ 9976.446199] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9981.724515] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9987.902823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9988.156514] Lustre: lustre-OST0000: new disk, initializing [ 9988.159893] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9988.165152] Lustre: 243115:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9988.278417] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9988.287107] Lustre: Skipped 1 previous similar message [ 9989.485073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9989.499750] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9989.638175] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9994.697388] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [10003.976053] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10008.113455] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10008.309349] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10012.806579] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10013.112586] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10015.216730] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10015.217246] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10015.230762] LustreError: Skipped 1 previous similar message [10015.245466] Lustre: Skipped 1 previous similar message [10015.258810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10015.272288] Lustre: Skipped 1 previous similar message [10021.209902] Lustre: server umount lustre-OST0000 complete [10025.441552] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10025.455786] Lustre: Skipped 1 previous similar message [10031.689724] Lustre: server umount lustre-MDT0000 complete [10035.687232] LustreError: 241155:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10035.725757] LustreError: 241155:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10036.070642] LustreError: 241142:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446685 with bad export cookie 13953182370706374199 [10036.075735] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10036.083938] LustreError: 241142:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10036.392053] Lustre: server umount lustre-MDT0001 complete [10045.820965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10046.239597] LustreError: 245332:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10051.558138] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [10055.650781] LustreError: 245333:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10055.665935] LustreError: 245333:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10061.380763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10061.821155] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10061.836501] Lustre: Skipped 1 previous similar message [10067.712655] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [10075.642417] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10081.407725] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10088.332946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10094.873786] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [10103.445674] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10250.721415] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10250.730835] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10250.744619] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10250.752640] Lustre: Skipped 6 previous similar messages [10256.485965] Lustre: server umount lustre-OST0000 complete [10261.474452] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10274.272136] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10274.444481] Lustre: server umount lustre-MDT0000 complete [10278.284230] LustreError: 245312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446927 with bad export cookie 13953182370706375081 [10278.309573] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10278.601255] Lustre: server umount lustre-MDT0001 complete [10370.446277] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [10373.019439] Key type lgssc unregistered [10373.303769] LNet: 248939:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10373.320655] LNetError: 248939:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10373.342812] LNet: Removed LNI 192.168.204.124@tcp [10374.104150] Key type .llcrypt unregistered [10374.106277] Key type ._llcrypt unregistered [10398.383384] Key type ._llcrypt registered [10398.389496] Key type .llcrypt registered [10398.518553] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [10412.511864] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [10413.195638] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10413.217056] alg: No test for adler32 (adler32-zlib) [10414.299349] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [10414.579535] LNet: Added LNI 192.168.204.124@tcp [8/256/0/180] [10416.280238] Key type lgssc registered [10417.306912] Lustre: Echo OBD driver; http://www.lustre.org/ [10461.558294] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [10472.661149] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10472.680410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10473.936301] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10473.983658] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10474.039158] Lustre: lustre-MDT0000: new disk, initializing [10474.164059] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10474.192486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10478.616593] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [10489.011140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10489.167101] Lustre: 253005:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10489.283819] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10489.292154] Lustre: Skipped 1 previous similar message [10489.316743] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10489.397861] Lustre: lustre-MDT0001: new disk, initializing [10489.486354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10489.533206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10489.544662] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10494.622677] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [10499.892619] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10506.249949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10506.588440] Lustre: lustre-OST0000: new disk, initializing [10506.594697] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10506.605262] Lustre: 254640:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10506.729799] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10510.268718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10510.275565] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10510.401644] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10512.866347] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [10524.451035] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10524.699970] Lustre: lustre-OST0001: new disk, initializing [10524.711734] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10524.728495] Lustre: 255510:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10524.907852] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10531.827942] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [10534.525724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10534.535867] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10534.600390] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10543.829806] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10551.442275] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10563.555540] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10563.574487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10563.589736] Lustre: Skipped 2 previous similar messages [10563.594187] Lustre: Skipped 1 previous similar message [10568.677279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10568.700819] Lustre: Skipped 3 previous similar messages [10573.795269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10573.813859] Lustre: Skipped 3 previous similar messages [10577.376983] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10577.719381] Lustre: server umount lustre-MDT0000 complete [10581.650871] LustreError: 252998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447231 with bad export cookie 2353150339344821405 [10581.654654] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10581.662195] LustreError: 252998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10582.118879] Lustre: server umount lustre-MDT0001 complete [10596.583450] Lustre: server umount lustre-OST0000 complete [10610.534821] Lustre: server umount lustre-OST0001 complete [10624.426132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10633.839758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10642.198613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10651.909991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10670.261379] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [10679.364180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10679.382646] Lustre: lustre-MDT0000: reset Object Index mappings [10679.736246] LustreError: 260525:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10679.755972] LustreError: 260525:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10679.804174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10684.433923] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [10684.903781] LustreError: 260526:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10690.020243] LustreError: 260525:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10692.725000] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10692.752238] Lustre: lustre-MDT0001: reset Object Index mappings [10693.251930] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10697.023490] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [10700.174321] Lustre: 261668:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10706.756131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10713.155967] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [10718.448064] LustreError: 262023:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10720.786427] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10721.069260] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10721.075056] Lustre: Skipped 1 previous similar message [10725.175754] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10725.184256] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10727.122820] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [10733.897854] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10737.087735] Lustre: 263537:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10743.684198] Lustre: server umount lustre-MDT0000 complete [10744.289874] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10744.301583] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10744.315639] Lustre: Skipped 1 previous similar message [10744.318414] LustreError: 260526:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10745.828090] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10745.843119] Lustre: Skipped 1 previous similar message [10747.406246] LustreError: 260506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447396 with bad export cookie 2353150339344828195 [10747.418982] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10747.424923] LustreError: 260506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10747.698638] Lustre: server umount lustre-MDT0001 complete [10762.038796] Lustre: server umount lustre-OST0000 complete [10776.192484] Lustre: server umount lustre-OST0001 complete [10797.747719] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 10:57:25 (1788447445) [10815.547791] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [10828.607699] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [10870.072684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10870.318201] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10870.355400] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10870.447571] Lustre: lustre-MDT0000: new disk, initializing [10870.507333] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10870.534904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10876.162552] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [10887.741265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10887.885243] Lustre: 267796:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10887.938504] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10887.942022] Lustre: Skipped 1 previous similar message [10888.024950] Lustre: lustre-MDT0001: new disk, initializing [10888.143151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10888.156216] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10892.336084] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [10901.093665] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10906.627504] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10914.691355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10915.014544] Lustre: lustre-OST0000: new disk, initializing [10915.017837] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10915.027151] Lustre: 269764:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10916.427048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10916.442789] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10916.632218] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10922.402179] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [10932.019825] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10936.049573] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10936.221976] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10940.954901] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10941.174318] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10947.554800] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10947.559043] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10947.563137] LustreError: Skipped 1 previous similar message [10947.570529] Lustre: Skipped 1 previous similar message [10947.572436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10947.583058] Lustre: Skipped 1 previous similar message [10949.406769] Lustre: server umount lustre-OST0000 complete [10954.721579] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10954.741230] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10954.753290] Lustre: Skipped 1 previous similar message [10957.793068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10957.803823] Lustre: Skipped 2 previous similar messages [10959.379510] Lustre: server umount lustre-MDT0000 complete [10962.914108] LustreError: 267802:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10962.929715] LustreError: 267802:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10963.462625] LustreError: 267788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447612 with bad export cookie 2353150339344830288 [10963.476805] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10963.483734] LustreError: 267788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10963.814817] Lustre: server umount lustre-MDT0001 complete [10982.344853] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [10994.037825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10994.454087] LustreError: 272701:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10994.511323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10994.515841] Lustre: Skipped 2 previous similar messages [11001.191912] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [11010.728647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11016.716251] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [11020.944457] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11028.142859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11036.013145] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [11045.709333] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11045.835844] Lustre: lustre-OST0001: new disk, initializing [11045.842107] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11045.847238] Lustre: 275021:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11048.031464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11048.043305] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11048.260995] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11053.813970] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [11062.832888] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11066.358858] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11072.441639] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [11072.529091] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [11082.721764] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11082.732094] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11082.752130] Lustre: Skipped 1 previous similar message [11082.760788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11084.178419] Lustre: server umount lustre-MDT0000 complete [11084.268655] LustreError: 272702:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11084.295964] LustreError: 272702:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11088.533682] LustreError: 272681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447738 with bad export cookie 2353150339344831170 [11088.543835] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11088.554450] LustreError: 272681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11088.954447] Lustre: server umount lustre-MDT0001 complete [11103.162417] Lustre: server umount lustre-OST0000 complete [11105.056198] Lustre: 250514:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788447738/real 1788447738] req@ffff913647ec0e00 x1875322668815616/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788447754 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11105.078082] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11105.089609] Lustre: Skipped 3 previous similar messages [11107.336323] Lustre: server umount lustre-OST0001 complete [11126.062860] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [11135.246669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11135.583451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11135.588849] Lustre: Skipped 3 previous similar messages [11139.687923] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [11149.676351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11155.294158] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [11158.776536] Lustre: 279506:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11165.827624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11173.906911] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [11175.342726] LustreError: 279861:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11175.352784] LustreError: 279861:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11181.574131] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [11182.945629] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11188.735083] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [11190.043436] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [11197.230829] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11200.384638] Lustre: 281380:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11205.564195] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [11205.701038] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [11217.524182] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 11:04:25 (1788447865) [11232.225380] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11232.231664] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11232.245036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11237.727447] Lustre: server umount lustre-MDT0000 complete [11241.498572] LustreError: 278344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447891 with bad export cookie 2353150339344837589 [11241.503662] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11241.514233] LustreError: 278344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11247.976990] Lustre: server umount lustre-MDT0001 complete [11257.973588] Lustre: server umount lustre-OST0000 complete [11261.751855] Lustre: server umount lustre-OST0001 complete [11269.094912] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [11280.448271] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [11333.818444] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [11347.125208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11347.476617] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11347.557615] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11347.687808] Lustre: lustre-MDT0000: new disk, initializing [11347.838135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11353.037146] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [11364.548275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11364.660531] Lustre: 285909:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11364.874313] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11369.261475] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [11381.019652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11381.278744] Lustre: 287543:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11382.928250] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11388.016776] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [11399.004406] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11399.223424] Lustre: 288414:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11399.301948] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11399.312896] Lustre: Skipped 6 previous similar messages [11400.590708] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11400.595890] Lustre: Skipped 1 previous similar message [11400.715706] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11405.029691] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [11413.820405] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11416.926744] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11416.932614] Lustre: Skipped 1 previous similar message [11432.049203] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 11:07:59 (1788448079) [11436.519451] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11436.532095] Lustre: Skipped 8 previous similar messages [11436.535141] LustreError: 285917:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11436.558837] LustreError: 285917:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [11436.623580] Lustre: server umount lustre-MDT0000 complete [11439.770920] LustreError: 285900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448089 with bad export cookie 2353150339344845520 [11439.773514] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11439.784407] LustreError: 285900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11440.001743] Lustre: server umount lustre-MDT0001 complete [11453.643510] Lustre: server umount lustre-OST0000 complete [11457.824131] Lustre: 250513:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788448091/real 1788448091] req@ffff913745a01180 x1875322668966016/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788448107 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11458.627663] Lustre: server umount lustre-OST0001 complete [11469.268718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11475.516679] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [11484.087852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11489.102896] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [11497.578184] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11504.313823] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11512.596313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11524.971593] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [11537.769767] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11554.267402] loop1: detected capacity change from 0 to 204800000 [11554.519611] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11554.535289] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11554.565702] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11559.564675] loop1: detected capacity change from 0 to 204800000 [11565.022930] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11565.159858] LustreError: 290649:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs3570: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11565.170246] LustreError: 290649:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs3570 [11565.175128] LustreError: 290649:0:(mgs_handler.c:517:mgs_target_reg()) fs3570-MDT0001: Error checking target index: rc = -13 [11565.182648] LustreError: fs3570-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11565.188973] LustreError: 294373:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -13 [11565.197125] LustreError: 294373:0:(tgt_mount.c:2132:server_put_super()) no obd fs3570-MDT0001 [11565.201640] LustreError: 294373:0:(tgt_mount.c:116:server_deregister_mount()) fs3570-MDT0001 not registered [11565.237757] Lustre: server umount fs3570-MDT0001 complete [11565.240641] LustreError: 294373:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11575.456529] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11575.631939] Lustre: Setting parameter fs3570-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs3570-MDT0001 [11575.708223] Lustre: srv-fs3570-MDT0001: No data found on store. Initialize space. [11575.712492] Lustre: Skipped 4 previous similar messages [11575.829670] Lustre: fs3570-MDT0001: new disk, initializing [11575.832719] Lustre: Skipped 3 previous similar messages [11575.958589] Lustre: srv-fs3570-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11575.967843] Lustre: 294841:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs3570-MDT0001-mdtlov: can't create new llog: rc = -115 [11575.988828] LustreError: 294841:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs3570-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11580.878667] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [11588.009134] Lustre: server umount fs3570-MDT0001 complete [11595.761986] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11603.754626] loop1: detected capacity change from 0 to 204800000 [11610.369556] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11610.664900] Lustre: 296410:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs3570-MDT0001-mdtlov: can't create new llog: rc = -115 [11610.677897] LustreError: 296410:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs3570-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11615.275972] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [11619.508705] Lustre: server umount fs3570-MDT0001 complete [11630.777659] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.24@tcp (stopping) [11630.790271] Lustre: Skipped 8 previous similar messages [11631.593481] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11631.616553] Lustre: Skipped 3 previous similar messages [11635.724404] Lustre: server umount lustre-MDT0000 complete [11646.082773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11646.216342] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11652.081821] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11652.278144] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [11662.392231] loop1: detected capacity change from 0 to 204800000 [11669.980080] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11670.215406] Lustre: 298862:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs3570-MDT0001-mdtlov: can't create new llog: rc = -115 [11670.227291] LustreError: 298862:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs3570-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11674.488593] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [11679.271439] Lustre: server umount fs3570-MDT0001 complete [11693.054800] LustreError: 292681:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11693.080261] LustreError: 292681:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 110 previous similar messages [11708.388428] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11708.404140] LustreError: Skipped 2 previous similar messages [11718.119551] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11718.260221] Lustre: server umount lustre-OST0000 complete [11728.632389] Lustre: server umount lustre-MDT0000 complete [11733.109978] LustreError: 290641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448382 with bad export cookie 2353150339344848201 [11733.121561] LustreError: 290641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11733.129260] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11733.538439] Lustre: server umount lustre-MDT0001 complete [11745.914619] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [11749.004862] Key type lgssc unregistered [11749.319538] LNet: 301209:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11749.332610] LNetError: 301209:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11749.359086] LNet: Removed LNI 192.168.204.124@tcp [11750.237292] Key type .llcrypt unregistered [11750.243543] Key type ._llcrypt unregistered [11773.983883] Key type ._llcrypt registered [11773.985927] Key type .llcrypt registered [11774.108143] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [11775.538028] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11775.565789] alg: No test for adler32 (adler32-zlib) [11776.709621] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [11776.934316] LNet: Added LNI 192.168.204.124@tcp [8/256/0/180] [11778.616184] Key type lgssc registered [11779.815605] Lustre: Echo OBD driver; http://www.lustre.org/ [11787.578734] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11787.598835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11789.009207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11794.380614] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [11799.525106] LustreError: 302479:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11799.557301] LustreError: 302479:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11803.375034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11803.928361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11808.313266] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [11816.215969] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11822.083930] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11829.619370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11830.100229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11835.242256] LustreError: 304307:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11836.729785] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [11840.487143] LustreError: 304309:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11844.543353] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11845.603747] LustreError: 304308:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11851.911033] LustreError: 304983:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11851.929825] LustreError: 304983:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11862.166584] LustreError: 304307:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11862.196619] LustreError: 304307:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11869.857386] loop1: detected capacity change from 0 to 204800000 [11870.076490] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11870.098491] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11870.154977] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11877.836552] loop1: detected capacity change from 0 to 204800000 [11879.416656] LustreError: 304308:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11879.461157] LustreError: 304308:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [11884.828531] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11885.006544] Lustre: fs1852-OST0000: new disk, initializing [11885.012725] Lustre: srv-fs1852-OST0000: No data found on store. Initialize space. [11885.034170] Lustre: 306217:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11885.187717] Lustre: fs1852-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11892.965193] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [11903.102466] Lustre: server umount fs1852-OST0000 complete [11913.351906] LustreError: 304308:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11913.374927] LustreError: 304308:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 18 previous similar messages [11915.238273] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11916.965920] loop1: detected capacity change from 0 to 204800000 [11923.990900] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11924.420663] Lustre: fs1852-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11930.620173] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11932.639805] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [11943.315327] loop2: detected capacity change from 0 to 204800000 [11943.591168] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11943.606744] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11943.624514] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11950.732612] loop2: detected capacity change from 0 to 204800000 [11957.302663] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11957.406329] LustreError: fs1852-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11957.421337] LustreError: fs1852-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11957.432130] LustreError: 309261:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -13 [11957.436988] LustreError: 309261:0:(tgt_mount.c:2132:server_put_super()) no obd fs1852-OST0001 [11957.440389] LustreError: 309261:0:(tgt_mount.c:116:server_deregister_mount()) fs1852-OST0001 not registered [11957.465606] Lustre: server umount fs1852-OST0001 complete [11957.469103] LustreError: 309261:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11969.904597] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11970.119596] Lustre: fs1852-OST0001: new disk, initializing [11970.133188] Lustre: srv-fs1852-OST0001: No data found on store. Initialize space. [11970.146041] Lustre: 309812:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11970.242628] Lustre: fs1852-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11976.875705] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [11979.918604] LustreError: 304309:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11979.953220] LustreError: 304309:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [11986.483988] Lustre: server umount fs1852-OST0000 complete [11996.894241] Lustre: server umount fs1852-OST0001 complete [12010.180924] loop1: detected capacity change from 0 to 204800000 [12015.602801] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12015.857739] Lustre: fs1852-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12022.981448] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [12029.927520] loop2: detected capacity change from 0 to 204800000 [12036.518598] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12045.297557] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [12055.387459] Lustre: server umount fs1852-OST0000 complete [12065.205978] Lustre: server umount fs1852-OST0001 complete [12068.843730] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12079.079776] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12079.089898] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12079.106463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12079.107535] Lustre: Skipped 1 previous similar message [12079.129472] Lustre: Skipped 1 previous similar message [12084.225845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12084.228344] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12084.241448] Lustre: Skipped 1 previous similar message [12089.315167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12089.327850] Lustre: Skipped 1 previous similar message [12091.360319] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12091.636597] Lustre: server umount lustre-OST0000 complete [12099.554571] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12099.564718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12099.578331] Lustre: Skipped 1 previous similar message [12102.727308] Lustre: server umount lustre-MDT0000 complete [12106.743791] LustreError: 311062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448756 with bad export cookie 11381478747243040500 [12106.757039] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12106.777308] LustreError: 311062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12107.070556] Lustre: server umount lustre-MDT0001 complete [12119.777042] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [12123.223828] Key type lgssc unregistered [12123.605972] LNet: 315737:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12123.620932] LNetError: 315737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12123.637763] LNet: Removed LNI 192.168.204.124@tcp [12124.673214] Key type .llcrypt unregistered [12124.674961] Key type ._llcrypt unregistered [12157.713161] Key type ._llcrypt registered [12157.714447] Key type .llcrypt registered [12157.850137] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [12178.290447] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [12179.192442] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12179.224564] alg: No test for adler32 (adler32-zlib) [12180.301459] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12180.566873] LNet: Added LNI 192.168.204.124@tcp [8/256/0/180] [12182.256597] Key type lgssc registered [12183.215221] Lustre: Echo OBD driver; http://www.lustre.org/ [12219.898172] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12219.931875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12221.302206] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12221.353242] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12221.451030] Lustre: lustre-MDT0000: new disk, initializing [12221.595785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12221.628807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12226.899716] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [12238.382769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12238.480169] Lustre: 319602:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12238.529825] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12238.533809] Lustre: Skipped 1 previous similar message [12238.622853] Lustre: lustre-MDT0001: new disk, initializing [12238.775996] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12238.839575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12238.860796] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12245.213345] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [12257.594879] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12265.664440] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12274.094517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12274.451887] Lustre: lustre-OST0000: new disk, initializing [12274.458060] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12274.477047] Lustre: 321571:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12274.605040] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12280.395660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12280.419446] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12280.558302] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12281.448485] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [12291.440591] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12294.233608] Lustre: Mounted lustre-client - version 2.17.58_2_gee80f9b [12295.586606] Lustre: Unmounted lustre-client [12309.477137] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12309.480399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12309.504050] Lustre: Skipped 1 previous similar message [12309.521657] Lustre: Skipped 1 previous similar message [12312.722485] Lustre: server umount lustre-OST0000 complete [12319.719140] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12319.734100] Lustre: Skipped 1 previous similar message [12319.736319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12319.747865] Lustre: Skipped 1 previous similar message [12324.746582] Lustre: server umount lustre-MDT0000 complete [12328.807678] LustreError: 319592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448978 with bad export cookie 16550777822540506287 [12328.809854] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12328.824239] LustreError: 319592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12329.128702] Lustre: server umount lustre-MDT0001 complete [12342.702511] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [12345.739897] Key type lgssc unregistered [12350.178177] LNet: 323729:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12350.200341] LNetError: 323729:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12425.003670] LNet: Removed LNI 192.168.204.124@tcp [12426.006139] Key type .llcrypt unregistered [12426.010430] Key type ._llcrypt unregistered [12448.260191] Key type ._llcrypt registered [12448.264335] Key type .llcrypt registered [12448.373223] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [12469.360323] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [12470.918810] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12470.948962] alg: No test for adler32 (adler32-zlib) [12472.079933] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12472.434822] LNet: Added LNI 192.168.204.124@tcp [8/256/0/180] [12474.112222] Key type lgssc registered [12475.251472] Lustre: Echo OBD driver; http://www.lustre.org/ [12509.950612] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 11:25:57 (1788449157) [12517.544967] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12517.566559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12518.873425] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12518.910616] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12519.044152] Lustre: lustre-MDT0000: new disk, initializing [12519.201406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12519.232190] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12524.736068] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [12537.203518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12537.306429] Lustre: 326950:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12537.354308] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12537.359901] Lustre: Skipped 1 previous similar message [12537.425608] Lustre: lustre-MDT0001: new disk, initializing [12537.501111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12537.538388] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12537.560422] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12542.817789] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [12553.718666] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12561.027996] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12568.758766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12569.112673] Lustre: lustre-OST0000: new disk, initializing [12569.115826] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12569.128310] Lustre: 328918:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12569.236595] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12576.116362] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [12576.347294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12576.362904] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12576.481101] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12586.106783] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12602.340674] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12602.343733] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12602.345773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12602.361729] LustreError: Skipped 1 previous similar message [12602.397229] Lustre: Skipped 1 previous similar message [12607.462971] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12607.469036] Lustre: Skipped 2 previous similar messages [12607.677647] Lustre: server umount lustre-OST0000 complete [12612.578768] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12612.585268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12612.599527] Lustre: Skipped 1 previous similar message [12612.616735] Lustre: Skipped 1 previous similar message [12617.700686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12617.707302] Lustre: Skipped 1 previous similar message [12617.878826] Lustre: server umount lustre-MDT0000 complete [12622.003313] LustreError: 326942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449271 with bad export cookie 2676683052621581686 [12622.013959] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12622.025968] LustreError: 326942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12622.525784] Lustre: server umount lustre-MDT0001 complete [12635.382587] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [12638.723614] Key type lgssc unregistered [12639.055489] LNet: 330823:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12639.066867] LNetError: 330823:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12639.101165] LNet: Removed LNI 192.168.204.124@tcp [12640.293703] Key type .llcrypt unregistered [12640.297119] Key type ._llcrypt unregistered [12663.457359] Key type ._llcrypt registered [12663.459314] Key type .llcrypt registered [12663.595924] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [12664.570760] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12664.608291] alg: No test for adler32 (adler32-zlib) [12665.699376] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12665.874398] LNet: Added LNI 192.168.204.124@tcp [8/256/0/180] [12667.642894] Key type lgssc registered [12669.169223] Lustre: Echo OBD driver; http://www.lustre.org/ [12678.636736] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12678.658638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12680.287650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12685.615603] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [12690.411883] LustreError: 332095:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12690.438299] LustreError: 332095:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12695.539532] LustreError: 332094:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12695.778727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12696.211901] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12700.276557] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [12708.670751] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12715.068557] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12722.430145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12722.935451] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12730.925924] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [12739.183802] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12751.955534] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12753.839411] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12761.284304] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.24@tcp (stopping) [12762.099022] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12762.103528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12762.129937] Lustre: Skipped 1 previous similar message [12762.138615] Lustre: Skipped 2 previous similar messages [12766.346319] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.24@tcp (stopping) [12771.461448] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.24@tcp (stopping) [12771.465668] Lustre: Skipped 3 previous similar messages [12773.344358] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12773.495652] Lustre: server umount lustre-MDT0000 complete [12776.583053] LustreError: 332089:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12777.095538] LustreError: 332075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449426 with bad export cookie 14046764728502538727 [12777.098682] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12777.108089] LustreError: 332075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12777.446758] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12777.461734] Lustre: Skipped 1 previous similar message [12777.477836] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12777.485278] Lustre: Skipped 3 previous similar messages [12781.705171] LustreError: 332091:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12781.727156] LustreError: 332091:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12783.658425] Lustre: server umount lustre-MDT0001 complete [12787.502215] Lustre: server umount lustre-OST0000 complete [12796.393788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12802.059706] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [12804.739611] Lustre: 336281:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.124@tcp [12806.797521] Lustre: server umount MGS complete [12816.827767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12817.476334] LustreError: 336853:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12817.491413] LustreError: 336853:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12817.542485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12822.249886] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [12827.788607] LustreError: 336850:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12827.810238] LustreError: 336850:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12832.591226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12833.005131] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12838.883876] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [12847.985275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12848.523743] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12856.872290] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [12865.307912] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12867.582638] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12874.619996] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12876.573948] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12882.916835] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [12882.925908] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12882.940172] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12882.944697] Lustre: Skipped 2 previous similar messages [12884.453334] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12887.084268] Lustre: server umount lustre-OST0000 complete [12894.695462] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12894.716948] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12894.719165] Lustre: Skipped 2 previous similar messages [12897.312403] Lustre: server umount lustre-MDT0000 complete [12899.809415] LustreError: 336849:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12899.822482] LustreError: 336849:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12900.914603] LustreError: 336834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449550 with bad export cookie 14046764728502539567 [12900.949260] LustreError: 336834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12900.957059] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12901.376278] Lustre: server umount lustre-MDT0001 complete [12916.209561] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [12920.030637] Key type lgssc unregistered [12920.477673] LNet: 340292:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12920.491674] LNetError: 340292:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12920.511946] LNet: Removed LNI 192.168.204.124@tcp [12921.624317] Key type .llcrypt unregistered [12921.626077] Key type ._llcrypt unregistered [12940.216821] Key type ._llcrypt registered [12940.222451] Key type .llcrypt registered [12940.352606] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [12956.360258] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [12957.685911] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12957.783040] alg: No test for adler32 (adler32-zlib) [12958.941325] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12959.241689] LNet: Added LNI 192.168.204.124@tcp [8/256/0/180] [12960.936397] Key type lgssc registered [12962.191493] Lustre: Echo OBD driver; http://www.lustre.org/ [13001.160298] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13001.178476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13002.451493] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13002.495067] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13002.677807] Lustre: lustre-MDT0000: new disk, initializing [13002.854075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13002.908959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13010.518843] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [13028.704734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13028.858863] Lustre: 343450:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13029.003435] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13029.012432] Lustre: Skipped 1 previous similar message [13029.179745] Lustre: lustre-MDT0001: new disk, initializing [13029.353032] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13029.432233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13029.470764] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13035.316538] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [13047.127970] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13054.383416] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13063.226605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13063.511392] Lustre: lustre-OST0000: new disk, initializing [13063.514563] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13063.519277] Lustre: 345418:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13063.648828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13069.664761] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [13073.488257] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13073.509122] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13073.643759] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13079.743651] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13084.557431] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13084.819406] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13089.731452] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13089.963562] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13094.375443] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13094.380434] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13094.392259] LustreError: Skipped 1 previous similar message [13094.393406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13094.423367] Lustre: Skipped 1 previous similar message [13098.455867] Lustre: server umount lustre-OST0000 complete [13104.615646] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13104.637226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13104.647258] Lustre: Skipped 2 previous similar messages [13109.730867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13109.741234] Lustre: Skipped 1 previous similar message [13114.850920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13114.856207] Lustre: Skipped 1 previous similar message [13116.384235] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13116.630337] Lustre: server umount lustre-MDT0000 complete [13120.925924] LustreError: 343440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449770 with bad export cookie 7987136540337409658 [13120.928140] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13120.951552] LustreError: 343440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13121.470604] Lustre: server umount lustre-MDT0001 complete [13131.275849] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 11:36:19 (1788449779) [13149.284492] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [13162.718500] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [13200.616402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13200.860419] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13200.911134] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13200.971994] Lustre: lustre-MDT0000: new disk, initializing [13201.037652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13201.054263] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13205.423404] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [13215.291497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13215.465232] Lustre: 350180:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13215.550726] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13215.553851] Lustre: Skipped 1 previous similar message [13215.692581] Lustre: lustre-MDT0001: new disk, initializing [13215.825872] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13215.879366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13215.888527] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13220.298611] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [13229.329098] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13235.576369] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13242.644242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13242.891430] Lustre: lustre-OST0000: new disk, initializing [13242.896431] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13242.902470] Lustre: 352149:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13242.979965] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13248.853239] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [13249.571614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13249.587377] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13249.716153] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13258.180926] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13290.978888] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13290.987490] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13290.993371] Lustre: Skipped 1 previous similar message [13291.005405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13291.020409] Lustre: Skipped 1 previous similar message [13301.224105] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13301.236349] Lustre: Skipped 3 previous similar messages [13303.266224] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13303.477342] Lustre: server umount lustre-OST0000 complete [13307.875223] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13307.886411] LustreError: Skipped 1 previous similar message [13307.888692] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13313.769459] Lustre: server umount lustre-MDT0000 complete [13316.578295] LustreError: 350193:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13316.595647] LustreError: 350193:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13318.187621] LustreError: 350173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449967 with bad export cookie 7987136540337410540 [13318.189091] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13318.194950] LustreError: 350173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13318.593609] Lustre: server umount lustre-MDT0001 complete [13329.064810] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [13331.814416] Key type lgssc unregistered [13332.174967] LNet: 354301:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13332.184042] LNetError: 354301:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13332.211451] LNet: Removed LNI 192.168.204.124@tcp [13332.910846] Key type .llcrypt unregistered [13332.913400] Key type ._llcrypt unregistered [13352.698419] Key type ._llcrypt registered [13352.700659] Key type .llcrypt registered [13352.810261] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [13353.675694] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13353.711738] alg: No test for adler32 (adler32-zlib) [13354.885971] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [13355.144148] LNet: Added LNI 192.168.204.124@tcp [8/256/0/180] [13356.840312] Key type lgssc registered [13357.920629] Lustre: Echo OBD driver; http://www.lustre.org/ [13366.314908] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13366.331284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13367.853984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13373.373909] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [13378.028750] LustreError: 355573:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13378.063947] LustreError: 355573:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13382.727084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13383.131250] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13388.228984] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [13396.687489] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13403.296674] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13410.624130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13411.028314] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13417.874489] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [13426.148932] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13444.067987] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13444.076045] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13444.088718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13444.107195] Lustre: Skipped 1 previous similar message [13446.055824] Lustre: server umount lustre-OST0000 complete [13454.307703] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13454.314310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13454.328528] Lustre: Skipped 2 previous similar messages [13454.353460] Lustre: Skipped 1 previous similar message [13456.725460] Lustre: server umount lustre-MDT0000 complete [13459.435283] LustreError: 356300:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13459.475522] LustreError: 356300:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13461.278528] LustreError: 355552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450110 with bad export cookie 10820027415246950048 [13461.285557] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13461.298564] LustreError: 355552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13462.138168] Lustre: server umount lustre-MDT0001 complete [13475.731446] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [13479.628416] Key type lgssc unregistered [13480.081218] LNet: 361072:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13480.106296] LNetError: 361072:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13481.138562] LNet: Removed LNI 192.168.204.124@tcp [13482.262382] Key type .llcrypt unregistered [13482.269907] Key type ._llcrypt unregistered [13503.770803] Key type ._llcrypt registered [13503.776967] Key type .llcrypt registered [13503.932193] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [13505.491516] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13505.585421] alg: No test for adler32 (adler32-zlib) [13506.849657] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [13507.292791] LNet: Added LNI 192.168.204.124@tcp [8/256/0/180] [13509.120595] Key type lgssc registered [13510.729763] Lustre: Echo OBD driver; http://www.lustre.org/ [13520.927656] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13520.968698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13522.696155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13528.259446] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [13533.159091] LustreError: 364928:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13533.192254] LustreError: 364928:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13538.274876] LustreError: 364930:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13539.977932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13540.433260] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13545.897315] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [13555.027404] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13562.523269] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13571.958863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13572.552747] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13580.889797] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [13590.823382] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13604.321135] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13604.326549] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13604.340592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13604.352227] Lustre: Skipped 1 previous similar message [13604.361686] Lustre: Skipped 1 previous similar message [13609.460569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13609.470185] Lustre: Skipped 1 previous similar message [13610.571913] Lustre: server umount lustre-OST0000 complete [13614.566621] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13614.570269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13614.581215] Lustre: Skipped 1 previous similar message [13619.687156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13619.705877] Lustre: Skipped 1 previous similar message [13620.941069] Lustre: server umount lustre-MDT0000 complete [13624.803918] LustreError: 364922:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13624.821593] LustreError: 364922:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13624.856414] LustreError: 368345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450274 with bad export cookie 5598972975146914560 [13624.857837] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13624.874615] LustreError: 368345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13625.343315] Lustre: server umount lustre-MDT0001 complete [13640.195862] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [13644.537524] Key type lgssc unregistered [13645.228658] LNet: 372615:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13645.244691] LNetError: 372615:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13645.291397] LNet: Removed LNI 192.168.204.124@tcp [13646.849581] Key type .llcrypt unregistered [13646.851758] Key type ._llcrypt unregistered [13666.640296] Key type ._llcrypt registered [13666.649311] Key type .llcrypt registered [13666.817721] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [13668.131947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13668.180346] alg: No test for adler32 (adler32-zlib) [13669.302498] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [13669.673946] LNet: Added LNI 192.168.204.124@tcp [8/256/0/180] [13671.440609] Key type lgssc registered [13672.843223] Lustre: Echo OBD driver; http://www.lustre.org/ [13682.190752] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13682.210080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13683.822045] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13688.752466] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [13693.922786] LustreError: 376424:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13693.942994] LustreError: 376424:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13697.930111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13698.364976] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13703.112670] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [13712.036968] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13719.399433] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13727.931488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13728.434594] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13737.119244] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [13746.739032] Lustre: DEBUG MARKER: oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13760.482754] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13760.490443] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13760.502426] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13763.553421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13763.559385] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13763.561254] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13763.567351] Lustre: Skipped 1 previous similar message [13765.015996] Lustre: server umount lustre-OST0000 complete [13770.215393] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13770.230069] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13770.262995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13773.797560] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13773.801783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13778.913503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13778.917459] Lustre: Skipped 2 previous similar messages [13783.008501] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13783.192146] Lustre: server umount lustre-MDT0000 complete [13784.034305] LustreError: 376417:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13784.054141] LustreError: 376417:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13786.385491] LustreError: 376341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450435 with bad export cookie 7301779578129469887 [13786.397278] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13786.413283] LustreError: 376341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13786.893028] Lustre: server umount lustre-MDT0001 complete [13798.824543] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [13801.734921] Key type lgssc unregistered [13802.003575] LNet: 382312:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13802.021372] LNetError: 382312:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13802.036451] LNet: Removed LNI 192.168.204.124@tcp [13802.910136] Key type .llcrypt unregistered [13802.912203] Key type ._llcrypt unregistered [13912.389385] Key type ._llcrypt registered [13912.397690] Key type .llcrypt registered [13912.563162] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [13912.965421] Key type .llcrypt unregistered [13912.971390] Key type ._llcrypt unregistered [13999.852986] Key type ._llcrypt registered [13999.859560] Key type .llcrypt registered [14000.000340] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [14000.390306] Key type .llcrypt unregistered [14000.400327] Key type ._llcrypt unregistered [14033.716137] Key type ._llcrypt registered [14033.720306] Key type .llcrypt registered [14033.870429] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [14034.078869] Key type .llcrypt unregistered [14034.082059] Key type ._llcrypt unregistered [14067.609096] Key type ._llcrypt registered [14067.613287] Key type .llcrypt registered [14067.843435] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [14068.284602] Key type .llcrypt unregistered [14068.290421] Key type ._llcrypt unregistered [14150.811805] Key type ._llcrypt registered [14150.820944] Key type .llcrypt registered [14150.931354] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [14151.210590] Key type .llcrypt unregistered [14151.218563] Key type ._llcrypt unregistered [14176.230641] Key type ._llcrypt registered [14176.233816] Key type .llcrypt registered [14176.349305] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [14176.602729] Key type .llcrypt unregistered [14176.605577] Key type ._llcrypt unregistered [14207.922628] Key type ._llcrypt registered [14207.933440] Key type .llcrypt registered [14208.110750] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [14208.462649] Key type .llcrypt unregistered [14208.467543] Key type ._llcrypt unregistered [14241.124644] Key type ._llcrypt registered [14241.130838] Key type .llcrypt registered [14241.254678] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [14241.679545] Key type .llcrypt unregistered [14241.686603] Key type ._llcrypt unregistered [14272.042837] Key type ._llcrypt registered [14272.046452] Key type .llcrypt registered [14272.169713] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [14272.443414] Key type .llcrypt unregistered [14272.447965] Key type ._llcrypt unregistered [14302.638387] Key type ._llcrypt registered [14302.642761] Key type .llcrypt registered [14302.765907] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [14302.991345] Key type .llcrypt unregistered [14302.997563] Key type ._llcrypt unregistered [14373.463782] Key type ._llcrypt registered [14373.470694] Key type .llcrypt registered [14373.602563] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [14373.974578] Key type .llcrypt unregistered [14373.978616] Key type ._llcrypt unregistered [14408.280053] Key type ._llcrypt registered [14408.282552] Key type .llcrypt registered [14408.371484] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [14408.622398] Key type .llcrypt unregistered [14408.623966] Key type ._llcrypt unregistered [14441.851524] Key type ._llcrypt registered [14441.856367] Key type .llcrypt registered [14441.941964] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [14442.315455] Key type .llcrypt unregistered [14442.321139] Key type ._llcrypt unregistered [14499.007739] Key type ._llcrypt registered [14499.014571] Key type .llcrypt registered [14499.168444] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [14514.874471] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [14515.686224] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14515.896910] alg: No test for adler32 (adler32-zlib) [14516.975421] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [14517.245507] LNet: Added LNI 192.168.204.124@tcp [8/256/0/180] [14518.936351] Key type lgssc registered [14520.104653] Lustre: Echo OBD driver; http://www.lustre.org/ [14561.455486] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14561.470238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14562.739385] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [14562.766553] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [14562.813488] Lustre: tst250fs-MDT0000: new disk, initializing [14562.989479] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14563.026215] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14568.600828] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [14580.661572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14580.803393] Lustre: 402819:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14580.858409] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [14580.863328] Lustre: Skipped 1 previous similar message [14580.940117] Lustre: tst250fs-MDT0001: new disk, initializing [14581.067620] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14581.129913] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14581.142686] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14586.851102] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [14591.943743] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [14599.526633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14599.804327] Lustre: tst250fs-OST0000: new disk, initializing [14599.809929] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [14599.817094] Lustre: 404449:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14599.914822] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14604.853814] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14604.862392] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14604.965382] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14605.810476] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [14617.231697] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14617.433693] Lustre: tst250fs-OST0001: new disk, initializing [14617.437673] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [14617.444871] Lustre: 405333:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14617.572617] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14624.774687] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_default_debug -1 all [14627.390460] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [14627.404809] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14627.507996] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [14630.729404] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [14633.572772] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [14633.579335] Lustre: Skipped 2 previous similar messages [14653.414810] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14653.435870] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14658.534767] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14658.546591] Lustre: Skipped 6 previous similar messages [14659.721264] Lustre: server umount tst250fs-MDT0000 complete [14663.866548] LustreError: 402811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451313 with bad export cookie 16926189725146738350 [14663.868918] LustreError: MGC192.168.204.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14663.879127] LustreError: 402811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14664.523657] Lustre: server umount tst250fs-MDT0001 complete [14679.327711] Lustre: server umount tst250fs-OST0000 complete [14693.544435] Lustre: server umount tst250fs-OST0001 complete [14716.030752] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [14725.670541] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [14762.770619] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 12:03:30 (1788451410) [14764.855404] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [14782.533654] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing set_hostid [14798.543690] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing load_modules_local [14835.149394] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14482 sec ========= 12:04:42 (1788451482) [14837.053646] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:04:44 (1788451484) === [14840.898454] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:04:48 (1788451488) === [14887.244510] Lustre: DEBUG MARKER: oleg424-server.virtnet: executing unload_modules_local [14890.604364] Key type lgssc unregistered [14890.938571] LNet: 414503:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14890.955181] LNetError: 414503:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14892.009156] LNet: Removed LNI 192.168.204.124@tcp [14893.037151] Key type .llcrypt unregistered [14893.039661] Key type ._llcrypt unregistered