[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 512564588 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003159] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.008377] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009026] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010012] pid_max: default: 32768 minimum: 301 [ 0.011137] LSM: Security Framework initializing [ 0.012070] Yama: becoming mindful. [ 0.013045] SELinux: Initializing. [ 0.014093] *** VALIDATE selinux *** [ 0.022618] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027591] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028178] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030118] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032117] *** VALIDATE tmpfs *** [ 0.034169] *** VALIDATE proc *** [ 0.035262] *** VALIDATE cgroup *** [ 0.036011] *** VALIDATE cgroup2 *** [ 0.037269] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039067] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041035] Spectre V2 : User space: Vulnerable [ 0.042010] Speculative Store Bypass: Vulnerable [ 0.045501] debug: unmapping init [mem 0xffffffffb6259000-0xffffffffb6260fff] [ 0.047168] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048526] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049020] ... version: 2 [ 0.049749] ... bit width: 48 [ 0.050012] ... generic registers: 4 [ 0.050802] ... value mask: 0000ffffffffffff [ 0.051023] ... max period: 00007fffffffffff [ 0.052026] ... fixed-purpose events: 3 [ 0.053015] ... event mask: 000000070000000f [ 0.054290] rcu: Hierarchical SRCU implementation. [ 0.056644] smp: Bringing up secondary CPUs ... [ 0.057679] x86: Booting SMP configuration: [ 0.058034] .... node #0, CPUs: #1 #2 #3 [ 0.061494] smp: Brought up 1 node, 4 CPUs [ 0.063013] smpboot: Max logical packages: 1 [ 0.064018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.145557] node 0 deferred pages initialised in 78ms [ 0.148123] devtmpfs: initialized [ 0.149290] x86/mm: Memory block size: 128MB [ 0.153353] gcov: version magic: 0x41383552 [ 0.155298] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.159137] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.162405] pinctrl core: initialized pinctrl subsystem [ 0.164178] [ 0.164821] ************************************************************* [ 0.167015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.169014] ** ** [ 0.171021] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.174014] ** ** [ 0.176015] ** This means that this kernel is built to expose internal ** [ 0.179013] ** IOMMU data structures, which may compromise security on ** [ 0.181011] ** your system. ** [ 0.183017] ** ** [ 0.186022] ** If you see this message and you are not debugging the ** [ 0.188013] ** kernel, report this immediately to your vendor! ** [ 0.191014] ** ** [ 0.193013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.196014] ************************************************************* [ 0.198637] NET: Registered protocol family 16 [ 0.201482] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.204089] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.207072] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.212115] cpuidle: using governor menu [ 0.213562] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.215453] PCI: Using configuration type 1 for base access [ 0.217142] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.225160] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.228110] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.232726] cryptd: max_cpu_qlen set to 1000 [ 0.235207] ACPI: Added _OSI(Module Device) [ 0.237013] ACPI: Added _OSI(Processor Device) [ 0.238019] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.240017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.245143] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.251400] ACPI: Interpreter enabled [ 0.253062] ACPI: PM: (supports S0 S3 S4 S5) [ 0.254013] ACPI: Using IOAPIC for interrupt routing [ 0.256125] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.259412] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.269933] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.272057] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.275018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.278086] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.283292] acpiphp: Slot [2] registered [ 0.284091] acpiphp: Slot [5] registered [ 0.285048] acpiphp: Slot [6] registered [ 0.286096] acpiphp: Slot [7] registered [ 0.287060] acpiphp: Slot [8] registered [ 0.288015] acpiphp: Slot [9] registered [ 0.288844] acpiphp: Slot [10] registered [ 0.289067] acpiphp: Slot [3] registered [ 0.290061] acpiphp: Slot [4] registered [ 0.290880] acpiphp: Slot [11] registered [ 0.292053] acpiphp: Slot [12] registered [ 0.292834] acpiphp: Slot [13] registered [ 0.294087] acpiphp: Slot [14] registered [ 0.295068] acpiphp: Slot [15] registered [ 0.296030] acpiphp: Slot [16] registered [ 0.297113] acpiphp: Slot [17] registered [ 0.298079] acpiphp: Slot [18] registered [ 0.299273] acpiphp: Slot [19] registered [ 0.301099] acpiphp: Slot [20] registered [ 0.302087] acpiphp: Slot [21] registered [ 0.303007] acpiphp: Slot [22] registered [ 0.303872] acpiphp: Slot [23] registered [ 0.304079] acpiphp: Slot [24] registered [ 0.305124] acpiphp: Slot [25] registered [ 0.307078] acpiphp: Slot [26] registered [ 0.308068] acpiphp: Slot [27] registered [ 0.308945] acpiphp: Slot [28] registered [ 0.310096] acpiphp: Slot [29] registered [ 0.311070] acpiphp: Slot [30] registered [ 0.311973] acpiphp: Slot [31] registered [ 0.313080] PCI host bridge to bus 0000:00 [ 0.313947] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.315016] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.316018] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.318017] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.320025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.322016] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.324217] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.327028] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.331355] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.341015] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.348041] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.350022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.352031] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.354019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.356468] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.359594] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.362035] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.364561] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.368013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.377017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.381012] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.385051] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.398017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.415024] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.435017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.444595] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.455030] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.463020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.482020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.491905] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.498014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.503024] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.525023] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.541767] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.562022] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.583018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.603015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.615564] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.624018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.630017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.645019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.654521] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.660017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.666019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.681025] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.691746] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.694394] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.696371] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.699356] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.702271] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.706108] iommu: Default domain type: Passthrough [ 0.708474] SCSI subsystem initialized [ 0.710136] ACPI: bus type USB registered [ 0.711106] usbcore: registered new interface driver usbfs [ 0.713105] usbcore: registered new interface driver hub [ 0.715080] usbcore: registered new device driver usb [ 0.716178] pps_core: LinuxPPS API ver. 1 registered [ 0.718009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.721052] PTP clock support registered [ 0.722145] EDAC MC: Ver: 3.0.0 [ 0.724124] PCI: Using ACPI for IRQ routing [ 0.725515] NetLabel: Initializing [ 0.726009] NetLabel: domain hash size = 128 [ 0.728024] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.729083] NetLabel: unlabeled traffic allowed by default [ 0.732044] vgaarb: loaded [ 0.734313] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.736012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.743000] clocksource: Switched to clocksource kvm-clock [ 0.846537] VFS: Disk quotas dquot_6.6.0 [ 0.847997] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.849773] *** VALIDATE ramfs *** [ 0.850554] *** VALIDATE hugetlbfs *** [ 0.851625] pnp: PnP ACPI init [ 0.853302] pnp: PnP ACPI: found 6 devices [ 0.870307] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.873689] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.876015] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.878348] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.880715] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.882901] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.885790] NET: Registered protocol family 2 [ 0.888224] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.893095] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.896830] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.902111] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.905673] TCP: Hash tables configured (established 65536 bind 65536) [ 0.908544] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.911807] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.914622] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.917922] NET: Registered protocol family 1 [ 0.921340] RPC: Registered named UNIX socket transport module. [ 0.923609] RPC: Registered udp transport module. [ 0.925507] RPC: Registered tcp transport module. [ 0.927469] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.930027] NET: Registered protocol family 44 [ 0.931524] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.933323] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.935023] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.936716] PCI: CLS 0 bytes, default 64 [ 0.938705] Unpacking initramfs... [ 2.284556] debug: unmapping init [mem 0xffffa012bcc54000-0xffffa012bffbffff] [ 2.287488] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.289874] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.293043] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.764714] Initialise system trusted keyrings [ 2.766309] Key type blacklist registered [ 2.767678] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.776180] zbud: loaded [ 2.779137] *** VALIDATE nfs *** [ 2.780425] *** VALIDATE nfs4 *** [ 2.781844] pstore: using deflate compression [ 2.785671] Platform Keyring initialized [ 2.877781] NET: Registered protocol family 38 [ 2.879176] Key type asymmetric registered [ 2.879950] Asymmetric key parser 'x509' registered [ 2.881060] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.883239] io scheduler mq-deadline registered [ 2.884500] io scheduler kyber registered [ 2.885512] io scheduler bfq registered [ 2.886634] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.888511] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.890263] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.892268] ACPI: Power Button [PWRF] [ 2.896333] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.900909] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.907751] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.912281] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.921115] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.946049] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.972692] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.976663] Non-volatile memory driver v1.3 [ 2.977938] Linux agpgart interface v0.103 [ 3.000686] virtio_blk virtio1: [vda] 145896 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.002269] vda: detected capacity change from 0 to 74698752 [ 3.019914] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.022417] vdb: detected capacity change from 0 to 1073741824 [ 3.036808] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.040177] vdc: detected capacity change from 0 to 2621440000 [ 3.055934] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.059096] vdd: detected capacity change from 0 to 2621440000 [ 3.086083] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.089414] vde: detected capacity change from 0 to 4294967296 [ 3.104617] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.107318] vdf: detected capacity change from 0 to 4294967296 [ 3.117031] libphy: Fixed MDIO Bus: probed [ 3.121921] usbcore: registered new interface driver usbserial_generic [ 3.123417] usbserial: USB Serial support registered for generic [ 3.124885] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.128149] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.129471] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.131100] mousedev: PS/2 mouse device common for all mice [ 3.133135] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.135474] rtc_cmos 00:05: RTC can wake from S4 [ 3.137689] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.139052] rtc_cmos 00:05: registered as rtc0 [ 3.141595] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.142385] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.146553] intel_pstate: CPU model not supported [ 3.148543] hid: raw HID events driver (C) Jiri Kosina [ 3.150561] usbcore: registered new interface driver usbhid [ 3.152263] usbhid: USB HID core driver [ 3.153537] drop_monitor: Initializing network drop monitor service [ 3.155323] Initializing XFRM netlink socket [ 3.156947] NET: Registered protocol family 10 [ 3.159521] Segment Routing with IPv6 [ 3.161092] NET: Registered protocol family 17 [ 3.162344] mpls_gso: MPLS GSO support [ 3.168223] RAS: Correctable Errors collector initialized. [ 3.170273] AVX version of gcm_enc/dec engaged. [ 3.172151] AES CTR mode by8 optimization enabled [ 3.235903] sched_clock: Marking stable (3235840115, 0)->(4150108348, -914268233) [ 3.238336] registered taskstats version 1 [ 3.239573] Loading compiled-in X.509 certificates [ 3.241175] zswap: loaded using pool lzo/zbud [ 3.260613] Key type big_key registered [ 3.272920] Key type encrypted registered [ 3.274522] ima: No TPM chip found, activating TPM-bypass! [ 3.276236] ima: Allocated hash algorithm: sha1 [ 3.277404] ima: No architecture policies found [ 3.278659] evm: Initialising EVM extended attributes: [ 3.279861] evm: security.selinux [ 3.280509] evm: security.ima [ 3.281236] evm: security.capability [ 3.281984] evm: HMAC attrs: 0x1 [ 3.283507] rtc_cmos 00:05: setting system clock to 2026-08-12 19:41:47 UTC (1786563707) [ 3.287894] debug: unmapping init [mem 0xffffffffb7203000-0xffffffffb73fffff] [ 3.289678] debug: unmapping init [mem 0xffffffffb5f82000-0xffffffffb6258fff] [ 3.296108] Write protecting the kernel read-only data: 28672k [ 3.298238] debug: unmapping init [mem 0xffffffffb4603000-0xffffffffb47fffff] [ 3.300015] debug: unmapping init [mem 0xffffffffb4f14000-0xffffffffb4ffffff] [ 3.325521] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.331738] systemd[1]: Detected virtualization kvm. [ 3.333137] systemd[1]: Detected architecture x86-64. [ 3.334727] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.360118] systemd[1]: No hostname configured. [ 3.362037] systemd[1]: Set hostname to . [ 3.364269] random: systemd: uninitialized urandom read (16 bytes read) [ 3.366918] systemd[1]: Initializing machine ID from random generator. [ 3.421469] random: ln: uninitialized urandom read (6 bytes read) [ 3.517802] random: systemd: uninitialized urandom read (16 bytes read) [ 3.521181] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.526467] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.535869] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. [ OK ] Reached target Swap. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.159854] device-mapper: uevent: version 1.0.3 [ 4.162126] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.753810] virtio_net virtio0 ens2: renamed from eth0 [ 4.763527] random: fast init done [ 4.846348] scsi host0: ata_piix [ 4.858677] scsi host1: ata_piix [ 4.860202] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.862046] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.078368] dracut-initqueue[588]: RTNETLINK answers: File exists [ 9.788245] random: crng init done [ 9.789836] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.072769] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.217357] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.484655] SELinux: Disabled at runtime. [ 11.546799] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.556276] systemd[1]: Detected virtualization kvm. [ 11.558153] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.058641] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.062041] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.066753] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.071350] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.075363] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.083407] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.093483] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Control Socket. Mounting Kernel Debug File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Listening on initctl Compatibility Named Pipe. Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ 12.231900] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-getty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting udev Coldplug all Devices... Mounting Huge Pages File System... [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.512177] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 12.906442] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.961333] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.073873] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.085368] EDAC sbridge: Ver: 1.1.2 [ 14.901578] Key type dns_resolver registered [ 15.216642] NFS: Registering the id_resolver key type [ 15.218469] Key type id_resolver registered [ 15.219660] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg242-server login: [ 27.434814] hrtimer: interrupt took 4804745 ns [ 86.221833] libcfs: loading out-of-tree module taints kernel. [ 86.414355] Key type ._llcrypt registered [ 86.422403] Key type .llcrypt registered [ 86.674320] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_hostid [ 109.955216] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 112.770136] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 112.819932] alg: No test for adler32 (adler32-zlib) [ 114.688382] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 115.964147] LNet: Added LNI 192.168.202.142@tcp [8/256/0/180] [ 117.911226] Key type lgssc registered [ 120.432828] Lustre: Echo OBD driver; http://www.lustre.org/ [ 146.613451] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 202.174937] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 218.749396] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 218.832952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 220.246460] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 220.299721] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 220.438159] Lustre: lustre-MDT0000: new disk, initializing [ 220.565146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 220.622841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 226.031254] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 241.934913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 242.093570] Lustre: 6522:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 242.133912] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 242.139418] Lustre: Skipped 1 previous similar message [ 242.271991] Lustre: lustre-MDT0001: new disk, initializing [ 242.384802] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 242.423338] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 242.435680] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 248.639931] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 253.860769] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 264.553259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 265.006087] Lustre: lustre-OST0000: new disk, initializing [ 265.008864] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 265.014851] Lustre: 8458:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 265.102172] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 272.276374] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 273.965615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 273.982413] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 274.083517] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 289.232540] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 289.365721] Lustre: lustre-OST0001: new disk, initializing [ 289.372753] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 289.377280] Lustre: 9531:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 289.447426] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 294.940458] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 294.969378] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 295.060484] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 295.911707] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 309.192098] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 321.170955] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 328.690298] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing check_logdir /tmp/testlogs/ [ 334.621209] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing yml_node [ 339.223903] Lustre: DEBUG MARKER: Client: 2.17.57.2 [ 342.323724] Lustre: DEBUG MARKER: MDS: 2.17.57.2 [ 345.628975] Lustre: DEBUG MARKER: OSS: 2.17.57.2 [ 347.346573] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Aug 12 15:47:30 EDT 2026 [ 366.673994] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 368.493098] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 386.017038] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 386.028503] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 386.043775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 387.043374] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 387.049557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 387.053357] Lustre: Skipped 2 previous similar messages [ 387.075331] Lustre: Skipped 3 previous similar messages [ 391.874204] Lustre: server umount lustre-MDT0000 complete [ 396.080140] LustreError: 7469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786564100 with bad export cookie 2728963185358401781 [ 396.082033] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 396.089156] LustreError: 7469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 397.285977] LustreError: 6529:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 397.290290] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 397.309101] LustreError: 6529:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 397.344114] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 401.312475] LustreError: 6527:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 401.312768] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 401.345373] Lustre: Skipped 2 previous similar messages [ 402.477574] Lustre: server umount lustre-MDT0001 complete [ 406.086190] Lustre: server umount lustre-OST0000 complete [ 409.954925] Lustre: server umount lustre-OST0001 complete [ 417.564712] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_hostid [ 428.422398] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 465.245795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 465.442809] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 465.466537] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 465.536944] Lustre: lustre-MDT0000: new disk, initializing [ 465.609468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 465.622745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 469.865253] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 480.572560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 480.761339] Lustre: 15085:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 480.881671] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 480.888781] Lustre: Skipped 1 previous similar message [ 480.999491] Lustre: lustre-MDT0001: new disk, initializing [ 481.138940] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 481.203293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 481.227729] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 486.474699] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 496.366834] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 501.794983] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 509.436964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 509.847839] Lustre: lustre-OST0000: new disk, initializing [ 509.853439] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 509.867291] Lustre: 17057:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 510.071612] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 511.097593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 511.118851] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 511.260130] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 517.818459] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 529.123617] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 534.683943] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 534.950406] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 539.556081] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 539.794945] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 542.208670] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 542.214107] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 542.217731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 542.248576] Lustre: Skipped 2 previous similar messages [ 542.288390] LustreError: Skipped 1 previous similar message [ 547.724898] Lustre: server umount lustre-OST0000 complete [ 552.417819] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 552.438341] Lustre: Skipped 1 previous similar message [ 552.439037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 552.476977] Lustre: Skipped 4 previous similar messages [ 566.241328] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 566.745528] Lustre: server umount lustre-MDT0000 complete [ 567.776797] LustreError: 15099:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 567.798355] LustreError: 15099:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 570.267983] LustreError: 15078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786564274 with bad export cookie 2728963185358403314 [ 570.282762] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 570.285651] LustreError: 15078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 570.579775] Lustre: server umount lustre-MDT0001 complete [ 577.597692] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 15:51:20 (1786564280) [ 598.022221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 598.486566] LustreError: 19934:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 598.596288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 602.827330] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 603.621301] LustreError: 19935:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 612.420687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 618.594591] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 627.552288] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 634.376424] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 637.126543] Lustre: Failing over lustre-MDT0000 [ 637.368918] Lustre: server umount lustre-MDT0000 complete [ 638.187485] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 638.205358] Lustre: Skipped 1 previous similar message [ 638.213681] LustreError: 20662:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 638.249738] LustreError: 20662:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 653.801066] Lustre: 3652:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786564342/real 1786564342] req@ffffa012037ebb80 x1873347944285696/t0(0) o400->MGC192.168.202.142@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786564358 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 653.826520] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 657.955868] LustreError: 19931:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 657.985868] LustreError: 19931:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 659.579994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 665.387698] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 665.395800] Lustre: Skipped 1 previous similar message [ 665.463909] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 669.709584] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 670.694997] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 670.719190] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 670.772698] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 683.148505] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 688.102498] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 688.115727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 688.130739] Lustre: Skipped 1 previous similar message [ 688.154701] Lustre: Skipped 5 previous similar messages [ 693.855403] Lustre: server umount lustre-MDT0000 complete [ 698.208961] LustreError: 19916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786564402 with bad export cookie 2728963185358404728 [ 698.210498] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 698.223778] LustreError: 19916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 698.736707] Lustre: server umount lustre-MDT0001 complete [ 708.529478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 709.225594] LustreError: 23276:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 709.243960] LustreError: 23276:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 714.766915] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 725.117122] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 730.957241] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 739.158554] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 745.478414] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 747.623557] Lustre: Failing over lustre-MDT0000 [ 747.638469] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 747.651270] Lustre: Skipped 1 previous similar message [ 747.652105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 747.656345] Lustre: Skipped 2 previous similar messages [ 747.808630] Lustre: server umount lustre-MDT0000 complete [ 765.929210] Lustre: 3652:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786564454/real 1786564454] req@ffffa01333d85500 x1873347944316928/t0(0) o400->MGC192.168.202.142@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786564470 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 765.967680] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 766.853620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 775.137364] LustreError: 23276:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 775.167940] LustreError: 23276:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [ 776.408481] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 781.194256] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 781.799755] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 781.801682] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 781.815614] Lustre: Skipped 1 previous similar message [ 781.834022] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 791.484484] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 797.152337] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 801.934245] Lustre: server umount lustre-MDT0000 complete [ 805.605636] LustreError: 23258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786564509 with bad export cookie 2728963185358405666 [ 805.621084] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 806.288370] Lustre: server umount lustre-MDT0001 complete [ 815.766400] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 15:55:18 (1786564518) [ 834.711222] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_hostid [ 846.441791] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 886.223888] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 897.709360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 898.084274] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 898.129870] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 898.250090] Lustre: lustre-MDT0000: new disk, initializing [ 898.319564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 898.324486] Lustre: Skipped 3 previous similar messages [ 898.337889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 903.279159] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 914.105050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 914.234685] Lustre: 29653:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 914.596484] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 919.473202] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 929.056661] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 935.346292] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 942.567695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 942.896923] Lustre: 31625:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 944.784938] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 944.829740] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 944.909994] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 944.925145] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 950.002431] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 961.126860] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 966.864807] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 991.204174] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 991.207655] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 991.214742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 991.226858] Lustre: Skipped 3 previous similar messages [ 991.238147] Lustre: Skipped 4 previous similar messages [ 992.921174] Lustre: server umount lustre-OST0000 complete [ 1001.441190] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1001.450304] LustreError: Skipped 1 previous similar message [ 1003.452592] Lustre: server umount lustre-MDT0000 complete [ 1006.566202] LustreError: 29662:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1006.579429] LustreError: 29662:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 1006.971530] LustreError: 31624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786564711 with bad export cookie 2728963185358406065 [ 1006.987484] LustreError: 31624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1006.989195] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1007.360451] Lustre: server umount lustre-MDT0001 complete [ 1019.552542] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 1024.004225] Key type lgssc unregistered [ 1024.239694] LNet: 33604:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1024.245069] LNetError: 33604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1024.259815] LNet: Removed LNI 192.168.202.142@tcp [ 1025.257226] Key type .llcrypt unregistered [ 1025.258761] Key type ._llcrypt unregistered [ 1056.051402] Key type ._llcrypt registered [ 1056.057378] Key type .llcrypt registered [ 1056.223771] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 1057.461050] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1057.683875] alg: No test for adler32 (adler32-zlib) [ 1058.686069] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 1058.914564] LNet: Added LNI 192.168.202.142@tcp [8/256/0/180] [ 1060.631179] Key type lgssc registered [ 1061.816959] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1070.447857] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1070.464732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1071.854092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1076.963209] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 1082.341298] LustreError: 35018:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1082.374758] LustreError: 35018:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1086.470525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1086.881663] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1092.621633] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 1100.756780] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1107.367524] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1114.466257] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1114.876599] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1121.428982] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 1122.297763] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1128.814633] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1131.329249] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1144.592704] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 16:00:47 (1786564847) [ 1146.933965] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1148.980169] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1157.999983] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 16:01:00 (1786564860) [ 1168.870952] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 16:01:11 (1786564871) [ 1170.343710] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1180.518273] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1180.526906] Lustre: Skipped 2 previous similar messages [ 1182.522863] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 16:01:25 (1786564885) [ 1184.564663] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1184.569580] Lustre: Skipped 2 previous similar messages [ 1193.488554] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1193.497756] Lustre: Skipped 4 previous similar messages [ 1209.435893] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 16:01:52 (1786564912) [ 1212.548315] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1216.813171] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1225.999465] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 16:02:08 (1786564928) [ 1228.283279] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1228.291386] Lustre: Skipped 6 previous similar messages [ 1242.822582] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 16:02:25 (1786564945) [ 1260.518153] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1260.529612] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1260.548130] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1264.608161] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1264.615649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1264.621792] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1264.649289] Lustre: Skipped 1 previous similar message [ 1269.733388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1269.756347] Lustre: Skipped 1 previous similar message [ 1273.823730] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1274.202794] Lustre: server umount lustre-OST0000 complete [ 1279.972136] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1279.980109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1279.983817] Lustre: Skipped 1 previous similar message [ 1279.993476] Lustre: Skipped 1 previous similar message [ 1284.760087] Lustre: server umount lustre-MDT0000 complete [ 1285.089691] LustreError: 35018:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1289.458860] LustreError: 35754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786564993 with bad export cookie 8449808309307181210 [ 1289.467527] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1289.480488] LustreError: 35754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1290.455989] Lustre: server umount lustre-MDT0001 complete [ 1302.583111] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 1306.102985] Key type lgssc unregistered [ 1306.460757] LNet: 42892:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1306.470450] LNetError: 42892:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1306.492821] LNet: Removed LNI 192.168.202.142@tcp [ 1307.433564] Key type .llcrypt unregistered [ 1307.439416] Key type ._llcrypt unregistered [ 1327.286595] Key type ._llcrypt registered [ 1327.297679] Key type .llcrypt registered [ 1327.482734] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_hostid [ 1342.982858] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 1344.437941] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1344.538867] alg: No test for adler32 (adler32-zlib) [ 1345.737397] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 1346.198192] LNet: Added LNI 192.168.202.142@tcp [8/256/0/180] [ 1347.919213] Key type lgssc registered [ 1349.627202] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1389.300609] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1389.323802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1390.664581] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1390.691939] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1390.753927] Lustre: lustre-MDT0000: new disk, initializing [ 1390.838237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1390.853430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1395.521138] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 1408.024412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1408.219056] Lustre: 46049:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1408.379061] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1408.397541] Lustre: Skipped 1 previous similar message [ 1408.542547] Lustre: lustre-MDT0001: new disk, initializing [ 1408.765459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1408.827797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1408.852039] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1414.448277] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 1424.074444] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1429.801113] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1437.141244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1437.562810] Lustre: lustre-OST0000: new disk, initializing [ 1437.566247] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1437.585341] Lustre: 48021:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1437.724511] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1445.093185] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 1446.553461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1446.579939] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1446.708935] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1454.884822] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1461.352261] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1461.711115] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1467.560758] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1467.872942] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1472.481711] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1472.485251] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1472.497450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1472.501683] Lustre: Skipped 1 previous similar message [ 1477.620152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1477.625857] Lustre: Skipped 2 previous similar messages [ 1482.727272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1482.746050] Lustre: Skipped 1 previous similar message [ 1484.255197] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1484.475853] Lustre: server umount lustre-OST0000 complete [ 1490.921127] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1490.929547] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1490.958796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1492.969174] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1498.083254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1498.092644] Lustre: Skipped 3 previous similar messages [ 1503.199212] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1503.380047] Lustre: server umount lustre-MDT0000 complete [ 1507.259131] LustreError: 46042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786565211 with bad export cookie 3152996545569520732 [ 1507.265961] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1507.272694] LustreError: 46042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1507.779990] Lustre: server umount lustre-MDT0001 complete [ 1518.350640] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 16:07:01 (1786565221) [ 1525.871220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1526.448251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1531.878283] LustreError: 50483:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1531.918656] LustreError: 50483:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1532.470077] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 1535.973482] LustreError: 50484:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1541.097954] LustreError: 50483:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1543.256763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1543.762456] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1549.520471] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 1558.055714] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1564.320879] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1572.187829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1572.750414] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1580.163775] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 1588.594980] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1603.338586] LustreError: 53033:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1604.286214] Lustre: Modifying parameter general.timeout=1 in log params [ 1605.591303] Lustre: Modifying parameter general.timeout=2 in log params [ 1608.378573] Lustre: Modifying parameter general.timeout=5 in log params [ 1608.385393] Lustre: Skipped 2 previous similar messages [ 1613.063694] Lustre: Modifying parameter general.timeout=10 in log params [ 1613.068742] Lustre: Skipped 4 previous similar messages [ 1621.385157] Lustre: Modifying parameter general.timeout=18 in log params [ 1621.395369] Lustre: Skipped 7 previous similar messages [ 1638.035152] Lustre: Modifying parameter general.timeout=36 in log params [ 1638.042570] Lustre: Skipped 17 previous similar messages [ 1671.069154] Lustre: Modifying parameter general.timeout=69 in log params [ 1671.080752] Lustre: Skipped 32 previous similar messages [ 1742.383280] Lustre: Disabling parameter general.timeout= in log params [ 1742.387203] Lustre: Skipped 60 previous similar messages [ 1746.132878] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 16:10:48 (1786565448) [ 1749.179180] Lustre: 59989:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1750.907374] Lustre: 60088:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1754.506528] Lustre: 60285:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1754.523957] Lustre: 60285:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1788.228812] Lustre: 61853:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1788.237815] Lustre: 61853:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1788.270833] LustreError: 61853:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1792.544285] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 16:11:35 (1786565495) [ 1806.828761] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1806.831380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1806.844434] Lustre: Skipped 2 previous similar messages [ 1806.865649] Lustre: Skipped 4 previous similar messages [ 1812.903210] Lustre: server umount lustre-MDT0000 complete [ 1817.060922] LustreError: 50484:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1817.706835] LustreError: 50465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786565521 with bad export cookie 3152996545569521614 [ 1817.714918] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1817.724835] LustreError: 50465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1818.035754] Lustre: server umount lustre-MDT0001 complete [ 1833.346673] Lustre: server umount lustre-OST0000 complete [ 1863.555877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1863.943343] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1864.240440] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1869.249726] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 1878.619081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1878.740528] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1878.772190] Lustre: 64386:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1884.614799] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 1893.181210] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1900.391899] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1908.710133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1908.974730] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1909.452087] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1909.454805] Lustre: Skipped 1 previous similar message [ 1916.468472] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 1924.965194] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1926.344218] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1926.351191] Lustre: Skipped 23 previous similar messages [ 1926.388318] Lustre: 66955:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1936.357978] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1936.381071] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1936.426257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1936.437831] Lustre: Skipped 3 previous similar messages [ 1939.320429] Lustre: server umount lustre-OST0000 complete [ 1945.568851] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1945.587054] LustreError: Skipped 1 previous similar message [ 1945.588715] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1945.600341] Lustre: Skipped 1 previous similar message [ 1949.536360] Lustre: server umount lustre-MDT0000 complete [ 1952.239259] LustreError: 64397:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1952.270064] LustreError: 64397:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1953.219649] LustreError: 64378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786565657 with bad export cookie 3152996545569524708 [ 1953.229863] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1953.232516] LustreError: 64378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1953.587691] Lustre: server umount lustre-MDT0001 complete [ 1964.917685] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 1968.399700] Key type lgssc unregistered [ 1968.773869] LNet: 68164:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1968.782517] LNetError: 68164:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1968.827698] LNet: Removed LNI 192.168.202.142@tcp [ 1969.720776] Key type .llcrypt unregistered [ 1969.723349] Key type ._llcrypt unregistered [ 1994.557577] Key type ._llcrypt registered [ 1994.560171] Key type .llcrypt registered [ 1994.715205] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 1996.421324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1996.472313] alg: No test for adler32 (adler32-zlib) [ 1997.710929] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 1997.997600] LNet: Added LNI 192.168.202.142@tcp [8/256/0/180] [ 1999.729320] Key type lgssc registered [ 2001.103124] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2008.724701] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2008.741625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2010.165581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2014.059782] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 2020.327959] LustreError: 69580:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2020.346829] LustreError: 69580:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2022.880774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2023.651870] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2028.828808] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 2036.197455] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2042.383572] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2049.338128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2049.874045] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2057.131444] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 2065.539079] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2076.628187] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2088.937728] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2088.947897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2088.956844] Lustre: Skipped 1 previous similar message [ 2088.984775] Lustre: Skipped 2 previous similar messages [ 2092.224458] Lustre: server umount lustre-MDT0000 complete [ 2094.055019] LustreError: 71887:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2094.075408] LustreError: 71887:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2096.351071] LustreError: 69560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786565800 with bad export cookie 3269635389898020778 [ 2096.353653] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2096.366726] LustreError: 69560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2096.727511] Lustre: server umount lustre-MDT0001 complete [ 2111.798146] Lustre: server umount lustre-OST0000 complete [ 2141.539478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2141.909911] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2141.941601] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2142.272601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2147.818311] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 2158.281608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2158.474962] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2158.565894] Lustre: 74456:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2159.244135] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2164.503085] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 2174.085303] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2181.790531] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2189.564237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2189.758815] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2190.112839] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2196.549222] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 2205.518798] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2207.633763] Lustre: 77029:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2209.786566] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2214.881882] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2214.894335] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2214.913121] Lustre: Skipped 1 previous similar message [ 2214.917601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2215.408579] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2220.525945] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2220.535809] Lustre: Skipped 2 previous similar messages [ 2221.090448] Lustre: server umount lustre-OST0000 complete [ 2225.632796] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2225.638618] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2225.649480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2225.649489] Lustre: Skipped 1 previous similar message [ 2225.679777] Lustre: Skipped 2 previous similar messages [ 2230.753102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2230.761875] Lustre: Skipped 1 previous similar message [ 2231.205441] Lustre: server umount lustre-MDT0000 complete [ 2234.815485] LustreError: 74448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786565939 with bad export cookie 3269635389898021625 [ 2234.820163] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2234.836124] LustreError: 74448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2235.233979] Lustre: server umount lustre-MDT0001 complete [ 2247.551703] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 2250.575807] Key type lgssc unregistered [ 2250.869875] LNet: 78241:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2250.890024] LNetError: 78241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2250.914954] LNet: Removed LNI 192.168.202.142@tcp [ 2252.108427] Key type .llcrypt unregistered [ 2252.110950] Key type ._llcrypt unregistered [ 2278.387353] Key type ._llcrypt registered [ 2278.391736] Key type .llcrypt registered [ 2278.559746] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 2280.088548] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2280.128957] alg: No test for adler32 (adler32-zlib) [ 2281.269069] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 2281.588760] LNet: Added LNI 192.168.202.142@tcp [8/256/0/180] [ 2283.359229] Key type lgssc registered [ 2284.806082] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2293.307987] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2293.348449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2294.862875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2299.994265] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 2305.005586] LustreError: 79653:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2305.021142] LustreError: 79653:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2308.510995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2308.960520] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2313.591841] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 2321.541420] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2326.749717] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2333.064866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2333.622622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2340.116783] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 2348.135681] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2355.407219] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2356.313181] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2358.248199] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2358.253315] Lustre: Skipped 1 previous similar message [ 2361.065288] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2361.069857] Lustre: Skipped 2 previous similar messages [ 2365.538394] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2365.548136] Lustre: Skipped 4 previous similar messages [ 2373.671957] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2373.678750] Lustre: Skipped 9 previous similar messages [ 2389.716988] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2389.726169] Lustre: Skipped 15 previous similar messages [ 2422.136565] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2422.141726] Lustre: Skipped 35 previous similar messages [ 2454.037922] Lustre: 87754:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2456.971733] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 16:22:39 (1786566159) [ 2458.522285] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2460.454870] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 16:22:43 (1786566163) [ 2466.784057] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2466.792568] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2466.823164] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2467.822819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2467.826928] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2467.831127] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2471.504805] Lustre: server umount lustre-OST0000 complete [ 2478.056061] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2478.062349] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2478.062516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2478.062521] Lustre: Skipped 1 previous similar message [ 2478.078464] Lustre: Skipped 1 previous similar message [ 2481.809662] Lustre: server umount lustre-MDT0000 complete [ 2483.175594] LustreError: 87865:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2483.195157] LustreError: 87865:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2485.541579] LustreError: 79634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786566189 with bad export cookie 16551068738450582191 [ 2485.553154] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2485.557160] LustreError: 79634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2485.912276] Lustre: server umount lustre-MDT0001 complete [ 2498.632772] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 2501.911534] Key type lgssc unregistered [ 2502.226772] LNet: 89009:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2502.233994] LNetError: 89009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2502.248641] LNet: Removed LNI 192.168.202.142@tcp [ 2502.959794] Key type .llcrypt unregistered [ 2502.962295] Key type ._llcrypt unregistered [ 2508.295209] Key type ._llcrypt registered [ 2508.299251] Key type .llcrypt registered [ 2508.399548] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2513.509729] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules [ 2515.172458] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2515.239313] alg: No test for adler32 (adler32-zlib) [ 2516.332562] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 2516.605994] LNet: Added LNI 192.168.202.142@tcp [8/256/0/180] [ 2518.343245] Key type lgssc registered [ 2519.690424] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2521.003991] LustreError: 90220:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2537.345512] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 2542.449699] LustreError: 90220:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2546.357147] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2546.371584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2548.044665] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2553.145746] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 2558.439025] LustreError: 91235:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2558.485738] LustreError: 91235:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2562.126288] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 16:24:25 (1786566265) [ 2563.565626] LustreError: 91236:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2567.180797] LustreError: 91268:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2567.187717] LustreError: 91268:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2567.195113] LustreError: 91268:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 2567.847298] Lustre: server umount lustre-MDT0000 complete [ 2582.351633] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 2585.236726] Key type lgssc unregistered [ 2585.484712] LNet: 92763:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2585.494192] LNetError: 92763:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2585.505233] LNet: Removed LNI 192.168.202.142@tcp [ 2586.205404] Key type .llcrypt unregistered [ 2586.208869] Key type ._llcrypt unregistered [ 2603.934288] Key type ._llcrypt registered [ 2603.936580] Key type .llcrypt registered [ 2604.041114] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 2605.382727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2605.444157] alg: No test for adler32 (adler32-zlib) [ 2606.607881] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 2606.847897] LNet: Added LNI 192.168.202.142@tcp [8/256/0/180] [ 2608.543498] Key type lgssc registered [ 2609.640704] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2617.746737] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2617.761476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2619.339602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2624.450884] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 2629.612711] LustreError: 93981:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2629.628209] LustreError: 93981:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2632.843281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2633.646706] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2638.129872] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 2645.561118] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2651.789730] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2657.812250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2658.269303] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2663.939390] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 2671.113662] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2717.920431] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 16:27:00 (1786566420) [ 2719.454205] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2721.224970] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 16:27:04 (1786566424) [ 2725.856940] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2725.876990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2725.888264] Lustre: Skipped 1 previous similar message [ 2730.464421] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2730.470787] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2730.484300] Lustre: Skipped 1 previous similar message [ 2730.492753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2731.668069] Lustre: server umount lustre-MDT0000 complete [ 2735.153871] LustreError: 93964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786566439 with bad export cookie 16740060624309159486 [ 2735.179962] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2735.526630] Lustre: server umount lustre-MDT0001 complete [ 2750.357175] Lustre: server umount lustre-OST0000 complete [ 2762.672378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2763.143272] LustreError: 98259:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2763.351528] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2768.076658] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 2768.353402] LustreError: 98260:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2773.482957] LustreError: 98259:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2776.934167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2777.829913] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2782.662598] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 2789.383176] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2794.753060] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2806.864245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2807.099370] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2807.107607] LustreError: 98248:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2807.130968] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2807.138924] LustreError: 100342:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2807.144537] LustreError: 100342:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2807.151165] LustreError: 100342:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2807.246529] Lustre: server umount lustre-OST0000 complete [ 2807.252254] LustreError: 100342:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2815.252656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2815.426983] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2815.429636] LustreError: 98247:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2815.435923] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2815.442553] LustreError: 100805:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2815.446498] LustreError: 100805:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2815.453399] LustreError: 100805:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2815.511899] Lustre: server umount lustre-OST0000 complete [ 2815.513706] LustreError: 100805:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2824.672090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2824.864648] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2824.869828] Lustre: Found index 0 for lustre-OST0000, updating log [ 2824.875101] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2824.951986] Lustre: lustre-OST0000: new disk, initializing [ 2824.956220] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2824.966051] Lustre: 101338:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2825.173109] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2832.199653] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 2836.253554] Lustre: Failing over lustre-OST0000 [ 2836.430736] Lustre: server umount lustre-OST0000 complete [ 2843.618243] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2843.626652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2843.632574] Lustre: Skipped 1 previous similar message [ 2843.654339] Lustre: Skipped 3 previous similar messages [ 2847.713467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2850.867381] Lustre: server umount lustre-MDT0000 complete [ 2852.836538] LustreError: 100734:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2852.857754] LustreError: 100734:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2854.181404] LustreError: 98240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786566558 with bad export cookie 16740060624309162230 [ 2854.183575] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2854.190393] LustreError: 98240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 2854.542865] Lustre: server umount lustre-MDT0001 complete [ 2863.545342] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 16:29:26 (1786566566) [ 2877.688787] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 2885.381481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2885.655285] LustreError: 103682:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2885.742731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2889.082381] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 2896.275126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2900.575675] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 2903.131921] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2908.344096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2908.924229] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2909.928081] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2909.952881] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2909.960335] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2909.970797] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2914.182091] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 2921.466560] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2921.544567] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2921.598789] Lustre: lustre-OST0001: new disk, initializing [ 2921.601861] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2921.606388] Lustre: 106000:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2921.825722] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2921.829806] Lustre: Skipped 2 previous similar messages [ 2927.238036] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 2927.265798] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2927.277813] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2927.382061] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2934.064576] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2938.822606] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2942.946375] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2942.951578] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2942.963633] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2942.964394] Lustre: Skipped 1 previous similar message [ 2942.980745] Lustre: Skipped 3 previous similar messages [ 2948.500492] Lustre: server umount lustre-MDT0001 complete [ 2950.175185] LustreError: 105193:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2950.189809] LustreError: 105193:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2955.657758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2955.764616] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2955.768488] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2955.829425] Lustre: 103670:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2957.296640] LustreError: 107575:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2957.301331] LustreError: 107575:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2957.307704] LustreError: 103675:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2957.356782] LustreError: 107580:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.142@tcp: cfg command failed: rc = -17 [ 2957.362791] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2957.362791] [ 2959.847264] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 2961.917462] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2966.797935] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 16:31:10 (1786566670) [ 2979.552574] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 2991.251570] Lustre: 109608:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3009.951828] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3012.852247] Lustre: 110742:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3079.137387] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3079.140882] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3079.147687] Lustre: Skipped 1 previous similar message [ 3079.150466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3079.153524] Lustre: Skipped 4 previous similar messages [ 3085.332265] Lustre: server umount lustre-MDT0000 complete [ 3087.616521] LustreError: 103663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786566791 with bad export cookie 16740060624309163161 [ 3087.618949] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3087.621265] LustreError: 103663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3087.775506] Lustre: server umount lustre-MDT0001 complete [ 3100.400225] Lustre: server umount lustre-OST0000 complete [ 3113.630846] Lustre: server umount lustre-OST0001 complete [ 3117.302914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3123.470571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3129.095287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3135.736777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3147.370219] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 3153.014044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3153.027760] Lustre: lustre-MDT0000: reset Object Index mappings [ 3153.290287] LustreError: 114699:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3153.298155] LustreError: 114699:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3153.354175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3153.357039] Lustre: Skipped 1 previous similar message [ 3155.081674] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 3159.199656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3159.213372] Lustre: lustre-MDT0001: reset Object Index mappings [ 3161.445130] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 3163.016631] Lustre: 115845:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3166.611026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3169.076960] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 3170.039173] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 3174.761287] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3178.160467] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:381 to 0x280000401:418) [ 3178.166623] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3178.225364] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3178.885520] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 3183.784462] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3185.702658] Lustre: 117721:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3216.476708] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 16:35:20 (1786566920) [ 3245.029444] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3245.031113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3245.036152] Lustre: Skipped 6 previous similar messages [ 3245.038653] Lustre: Skipped 9 previous similar messages [ 3250.635056] Lustre: server umount lustre-MDT0000 complete [ 3252.852987] LustreError: 114681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786566957 with bad export cookie 16740060624309210236 [ 3252.860699] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3252.864105] LustreError: 114681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3255.265892] LustreError: 114694:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3255.286709] LustreError: 114694:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3259.229870] Lustre: server umount lustre-MDT0001 complete [ 3267.945133] Lustre: server umount lustre-OST0000 complete [ 3278.779739] Lustre: server umount lustre-OST0001 complete [ 3287.705032] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_hostid [ 3304.534801] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 3356.023581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3356.294614] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3356.329830] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3356.411319] Lustre: lustre-MDT0000: new disk, initializing [ 3356.496925] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3356.500079] Lustre: Skipped 3 previous similar messages [ 3356.510262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3361.098343] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 3367.264624] Lustre: server umount lustre-MDT0000 complete [ 3375.796800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3376.060736] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3376.072061] Lustre: 122010:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3376.083662] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3380.629935] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 3384.896126] Lustre: server umount lustre-MDT0000 complete [ 3391.607531] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 16:38:14 (1786567094) [ 3393.047445] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3394.638762] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 16:38:17 (1786567097) [ 3410.982470] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 3421.805834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3426.582495] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 3435.074988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3435.202091] Lustre: 124058:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3435.256694] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3435.260159] Lustre: Skipped 1 previous similar message [ 3435.343612] Lustre: lustre-MDT0001: new disk, initializing [ 3435.551084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3435.567936] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3440.385972] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 3444.977319] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3451.300156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3451.590335] Lustre: lustre-OST0000: new disk, initializing [ 3451.595417] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3451.600185] Lustre: 125634:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3452.802530] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3452.821549] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3452.995330] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3458.265339] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 3468.140791] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3468.334866] Lustre: lustre-OST0001: new disk, initializing [ 3468.338142] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3468.343886] Lustre: 126506:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3470.161302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3470.175805] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3470.283609] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3473.483982] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 3482.291087] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3485.578343] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3864.202277] Lustre: 132916:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3864.213255] Lustre: 132916:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3864.220092] Lustre: 132916:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3864.228940] Lustre: 132916:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3864.238570] Lustre: 132916:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3864.244560] Lustre: 132916:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4159.570262] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 16:51:01 (1786567861) [ 4176.872168] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4176.886077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4176.892888] Lustre: Skipped 5 previous similar messages [ 4176.896790] Lustre: Skipped 10 previous similar messages [ 4181.424876] Lustre: server umount lustre-MDT0000 complete [ 4181.992299] LustreError: 127320:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4182.030515] LustreError: 127320:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4185.326736] LustreError: 132940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786567889 with bad export cookie 16740060624309272039 [ 4185.333221] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4185.345085] LustreError: 132940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4190.183533] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4190.188789] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4190.193087] Lustre: Skipped 1 previous similar message [ 4190.215111] Lustre: Skipped 2 previous similar messages [ 4191.997347] Lustre: server umount lustre-MDT0001 complete [ 4202.732699] Lustre: server umount lustre-OST0000 complete [ 4213.014254] Lustre: server umount lustre-OST0001 complete [ 4219.660335] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_hostid [ 4229.691898] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 4264.131896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4264.329970] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4264.393772] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4264.533106] Lustre: lustre-MDT0000: new disk, initializing [ 4264.702766] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4264.709213] Lustre: Skipped 5 previous similar messages [ 4264.733157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4269.622623] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 4280.828210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4280.959975] Lustre: 135977:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4281.025408] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4281.033770] Lustre: Skipped 1 previous similar message [ 4281.116886] Lustre: lustre-MDT0001: new disk, initializing [ 4281.285360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4281.326529] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4286.362472] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 4297.026476] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4303.435498] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4311.290592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4311.495249] Lustre: lustre-OST0000: new disk, initializing [ 4311.498582] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4311.504607] Lustre: 137948:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4311.625838] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4311.639497] Lustre: Skipped 1 previous similar message [ 4313.121755] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4313.142029] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4313.253964] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4318.612975] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 4328.836815] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4338.431836] Lustre: lustre-MDD0000: changelog on [ 4341.034746] Lustre: lustre-MDD0001: changelog on [ 5800.243207] Lustre: lustre-MDD0001: changelog off [ 5801.323258] Lustre: 137777:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5801.901759] Lustre: lustre-MDD0000: changelog off [ 5808.611544] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5808.618616] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5808.623601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5808.624119] LustreError: Skipped 1 previous similar message [ 5808.627916] Lustre: Skipped 1 previous similar message [ 5813.734267] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5813.743621] Lustre: Skipped 1 previous similar message [ 5818.849228] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5819.362076] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5819.486203] Lustre: server umount lustre-OST0000 complete [ 5822.435363] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5822.444598] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5822.450826] Lustre: Skipped 1 previous similar message [ 5823.968869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5823.973530] Lustre: Skipped 2 previous similar messages [ 5823.976373] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5827.548053] Lustre: server umount lustre-MDT0000 complete [ 5829.092752] LustreError: 135985:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5829.106865] LustreError: 135985:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 5829.611891] LustreError: 135971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786569533 with bad export cookie 16740060624310089772 [ 5829.619161] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5829.624050] LustreError: 135971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5829.820535] Lustre: server umount lustre-MDT0001 complete [ 5836.450074] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 5838.326628] Key type lgssc unregistered [ 5838.528726] LNet: 147028:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5838.535450] LNetError: 147028:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5838.548257] LNet: Removed LNI 192.168.202.142@tcp [ 5839.084137] Key type .llcrypt unregistered [ 5839.086205] Key type ._llcrypt unregistered [ 5851.587761] Key type ._llcrypt registered [ 5851.589946] Key type .llcrypt registered [ 5851.657376] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_hostid [ 5861.334618] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 5861.937460] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5861.950549] alg: No test for adler32 (adler32-zlib) [ 5862.891436] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 5863.067423] LNet: Added LNI 192.168.202.142@tcp [8/256/0/180] [ 5864.703217] Key type lgssc registered [ 5865.538343] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5886.273353] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5886.287432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5887.446241] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5887.486615] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5887.543532] Lustre: lustre-MDT0000: new disk, initializing [ 5887.584254] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5887.593593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5890.138923] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 5897.065980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5897.141755] Lustre: 150207:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5897.179583] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5897.182604] Lustre: Skipped 1 previous similar message [ 5897.220638] Lustre: lustre-MDT0001: new disk, initializing [ 5897.277779] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5897.301768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5897.307366] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5899.851605] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 5905.901946] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5909.460588] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5914.326440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5914.543560] Lustre: lustre-OST0000: new disk, initializing [ 5914.546950] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5914.549852] Lustre: 152173:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5914.597356] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5918.435085] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 5921.822752] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5921.833957] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5921.892316] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5924.568189] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5927.963597] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_module obdecho/obdecho [ 5929.207089] LustreError: 153175:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 5929.211133] LustreError: 153175:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 5930.259945] LustreError: 153272:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 5937.124588] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5937.125510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5937.137491] Lustre: Skipped 2 previous similar messages [ 5937.140776] Lustre: Skipped 1 previous similar message [ 5941.803333] Lustre: server umount lustre-MDT0000 complete [ 5943.894892] LustreError: 150197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786569648 with bad export cookie 6723606713048666014 [ 5943.898322] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5943.907137] LustreError: 150197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5944.113347] Lustre: server umount lustre-MDT0001 complete [ 5955.978909] Lustre: server umount lustre-OST0000 complete [ 5969.031411] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_hostid [ 5975.385800] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 5997.442492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5997.592863] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5997.612800] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5997.667270] Lustre: lustre-MDT0000: new disk, initializing [ 5997.719911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5997.733346] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6000.188992] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 6006.327306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6006.397214] Lustre: 156984:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6006.434933] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6006.437900] Lustre: Skipped 1 previous similar message [ 6006.481584] Lustre: lustre-MDT0001: new disk, initializing [ 6006.534860] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6006.558442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6006.566405] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6008.876384] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 6014.809350] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6018.034328] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6021.967820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6022.117428] Lustre: lustre-OST0000: new disk, initializing [ 6022.120275] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6022.124402] Lustre: 158948:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6022.175303] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6025.463175] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 6030.893542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6030.897533] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6030.973466] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6031.323904] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6033.900376] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6034.053351] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6036.614556] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6036.752792] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6041.569090] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6041.570072] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6041.572838] LustreError: Skipped 1 previous similar message [ 6041.584729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6041.593552] Lustre: Skipped 2 previous similar messages [ 6046.691845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6046.695911] Lustre: Skipped 1 previous similar message [ 6051.809075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6051.812582] Lustre: Skipped 1 previous similar message [ 6052.319127] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6052.424394] Lustre: server umount lustre-OST0000 complete [ 6056.929306] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6056.936855] Lustre: Skipped 2 previous similar messages [ 6056.937016] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6056.943020] Lustre: Skipped 1 previous similar message [ 6060.398424] Lustre: server umount lustre-MDT0000 complete [ 6062.048058] LustreError: 156995:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6062.061104] LustreError: 156995:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6062.296667] LustreError: 156974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786569766 with bad export cookie 6723606713048666973 [ 6062.302470] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6062.303776] LustreError: 156974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6062.460992] Lustre: server umount lustre-MDT0001 complete [ 6064.520953] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 17:22:48 (1786569768) [ 6065.535492] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6066.578292] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 17:22:50 (1786569770) [ 6070.950444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6071.230613] LustreError: 161359:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6071.262411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6073.720920] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 6076.384219] LustreError: 161360:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6078.607399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6080.988850] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 6084.581657] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6087.232046] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6090.772820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6093.523804] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 6097.047959] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6111.907515] Lustre: server umount lustre-MDT0000 complete [ 6112.806133] LustreError: 161355:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6113.727947] LustreError: 161339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786569817 with bad export cookie 6723606713048667855 [ 6113.732552] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6113.744780] LustreError: 161339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6113.761067] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6113.771872] Lustre: Skipped 1 previous similar message [ 6113.776957] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6117.915813] LustreError: 161356:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6117.930128] LustreError: 161356:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6128.095171] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6128.096081] LustreError: 163850:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6128.105386] LustreError: 163850:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6128.229490] Lustre: server umount lustre-MDT0001 complete [ 6132.318143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6139.349623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6139.484171] LustreError: 165218:0:(llog.c:1655:llog_backup()) MGC192.168.202.142@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6139.489592] Lustre: 165218:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.142@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6153.755262] LustreError: 165224:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6153.765418] LustreError: 165224:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6188.514114] LustreError: 165225:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6188.529485] LustreError: 165225:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 27 previous similar messages [ 6200.287415] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6200.290280] Lustre: 165218:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.202.142@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6241.266942] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6241.273694] Lustre: Skipped 2 previous similar messages [ 6244.061701] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 6246.381609] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6249.647203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6254.108575] LustreError: 165224:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6254.133394] LustreError: 165224:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 42 previous similar messages [ 6258.656312] LustreError: 148429:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000557a17b3 x1873353970398592/t0(0) o250->MGC192.168.202.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6261.814627] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 6264.304811] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6267.022702] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6268.108316] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6271.693706] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6272.610338] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6283.500649] Lustre: server umount lustre-MDT0000 complete [ 6284.769065] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6284.772665] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6284.781155] Lustre: Skipped 2 previous similar messages [ 6285.550191] LustreError: 161341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786569989 with bad export cookie 6723606713048674309 [ 6285.550730] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6285.559932] LustreError: 161341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6285.749321] Lustre: server umount lustre-MDT0001 complete [ 6298.423950] Lustre: server umount lustre-OST0000 complete [ 6313.120270] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_hostid [ 6320.099295] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 6345.276473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6345.435261] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6345.455473] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6345.504739] Lustre: lustre-MDT0000: new disk, initializing [ 6345.554959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6345.560898] Lustre: Skipped 1 previous similar message [ 6345.576787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6348.389710] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 6355.498698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6355.572383] Lustre: 170295:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6355.753671] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6358.398356] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 6364.466606] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6367.698131] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6371.963582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6372.121185] Lustre: 172263:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6376.717809] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 6382.113349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6382.119228] Lustre: Skipped 1 previous similar message [ 6382.121822] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6382.181659] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6383.449622] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6386.534468] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6386.752948] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6389.926219] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6390.089524] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6392.290243] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6392.291101] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6392.295652] LustreError: Skipped 1 previous similar message [ 6392.297405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6392.297412] Lustre: Skipped 5 previous similar messages [ 6392.324111] Lustre: Skipped 1 previous similar message [ 6397.572316] Lustre: server umount lustre-OST0000 complete [ 6402.026016] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6406.365541] Lustre: server umount lustre-MDT0000 complete [ 6407.651292] LustreError: 170308:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6407.666481] LustreError: 170308:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 6408.981493] LustreError: 170288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786570113 with bad export cookie 6723606713048691116 [ 6408.992886] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6408.996368] LustreError: 170288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6409.168698] Lustre: server umount lustre-MDT0001 complete [ 6414.517413] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 17:28:38 (1786570118) [ 6418.681930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6421.894116] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 6428.326864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6431.363950] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 6435.479494] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6439.508957] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6444.011690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6448.596199] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 6453.709313] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6476.372294] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 17:29:39 (1786570179) [ 6479.330493] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6479.335657] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6479.341407] Lustre: Skipped 2 previous similar messages [ 6479.344693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6479.351030] Lustre: Skipped 5 previous similar messages [ 6484.605224] Lustre: server umount lustre-OST0000 complete [ 6490.082631] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6490.087852] LustreError: Skipped 1 previous similar message [ 6501.343232] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6501.438672] Lustre: server umount lustre-MDT0000 complete [ 6503.834207] LustreError: 174655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786570208 with bad export cookie 6723606713048691998 [ 6503.837037] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6503.854474] LustreError: 174655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6504.104283] Lustre: server umount lustre-MDT0001 complete [ 6511.093402] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 6512.698565] Key type lgssc unregistered [ 6512.932573] LNet: 178424:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6512.938960] LNetError: 178424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6512.950601] LNet: Removed LNI 192.168.202.142@tcp [ 6513.473382] Key type .llcrypt unregistered [ 6513.475400] Key type ._llcrypt unregistered [ 6525.944444] Key type ._llcrypt registered [ 6525.947154] Key type .llcrypt registered [ 6526.029625] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 6526.726162] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6526.791075] alg: No test for adler32 (adler32-zlib) [ 6527.717888] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 6527.888801] LNet: Added LNI 192.168.202.142@tcp [8/256/0/180] [ 6529.535252] Key type lgssc registered [ 6530.264775] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6535.315459] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6535.331375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6550.815401] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6555.935386] LustreError: 179643:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.142@tcp: failed processing log, type 4: rc = -110 [ 6561.055609] LustreError: 179643:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6561.069806] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 6561.074205] LustreError: MGC192.168.202.142@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 6561.088247] LustreError: 179643:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 6561.098549] LustreError: 179643:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 6561.106422] LustreError: 179643:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 6561.141864] Lustre: server umount lustre-OST0000 complete [ 6561.144574] LustreError: 179643:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6566.556480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6567.822512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6570.386144] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 6575.792802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6576.023866] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6578.331546] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 6582.446419] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6585.681680] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6589.492391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6589.618528] LustreError: 181954:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6589.624747] Lustre: 181954:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.202.142@tcp: invalid local config log lustre-OST0000: rc = -22 [ 6589.717384] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6592.666976] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 6596.693154] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6598.113491] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6598.121106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6598.124258] Lustre: Skipped 2 previous similar messages [ 6598.140042] Lustre: Skipped 2 previous similar messages [ 6601.185186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6604.199941] Lustre: server umount lustre-MDT0000 complete [ 6606.234733] LustreError: 180122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786570310 with bad export cookie 9179875239890393779 [ 6606.238267] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6606.242763] LustreError: 180122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6606.305405] LustreError: 180136:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6606.305517] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6606.314284] LustreError: 180136:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6606.322508] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6606.322515] Lustre: Skipped 2 previous similar messages [ 6607.392562] LustreError: 180870:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000ece80026 x1873354667613824/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6607.468833] Lustre: server umount lustre-MDT0001 complete [ 6609.921291] Lustre: server umount lustre-OST0000 complete [ 6615.652357] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_hostid [ 6622.014687] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 6643.168702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6643.338578] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6643.362217] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6643.415318] Lustre: lustre-MDT0000: new disk, initializing [ 6643.462534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6643.477869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6645.800413] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 6652.240534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6652.331551] Lustre: 185476:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6652.376151] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6652.379061] Lustre: Skipped 1 previous similar message [ 6652.432679] Lustre: lustre-MDT0001: new disk, initializing [ 6652.494493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6652.509149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6652.516019] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6655.158564] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 6660.515256] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6663.043250] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6666.135494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6666.287888] Lustre: lustre-OST0000: new disk, initializing [ 6666.290422] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6666.293901] Lustre: 187442:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6666.336666] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6669.058737] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 6671.884388] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6671.889866] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6671.939107] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6674.036167] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6675.994773] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6676.095841] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6677.928876] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6678.020603] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6682.083029] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6682.085862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6682.091018] Lustre: Skipped 1 previous similar message [ 6682.091298] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6687.206620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6687.209453] Lustre: Skipped 2 previous similar messages [ 6693.343117] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6693.425755] Lustre: server umount lustre-OST0000 complete [ 6697.440881] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6697.444494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6697.447768] Lustre: Skipped 1 previous similar message [ 6697.450543] Lustre: Skipped 2 previous similar messages [ 6701.428809] Lustre: server umount lustre-MDT0000 complete [ 6702.560416] LustreError: 185489:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6702.570085] LustreError: 185489:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6703.534422] LustreError: 185470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786570407 with bad export cookie 9179875239890394507 [ 6703.539456] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6703.540936] LustreError: 185470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6703.728710] Lustre: server umount lustre-MDT0001 complete [ 6708.593652] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 17:33:32 (1786570412) [ 6718.942827] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 6720.555667] Key type lgssc unregistered [ 6720.742734] LNet: 190389:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6720.747602] LNetError: 190389:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6720.755661] LNet: Removed LNI 192.168.202.142@tcp [ 6721.269433] Key type .llcrypt unregistered [ 6721.275431] Key type ._llcrypt unregistered [ 6731.435749] Key type ._llcrypt registered [ 6731.437476] Key type .llcrypt registered [ 6731.513846] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 6732.022643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6732.074232] alg: No test for adler32 (adler32-zlib) [ 6733.005729] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 6733.118673] LNet: Added LNI 192.168.202.142@tcp [8/256/0/180] [ 6734.719711] Key type lgssc registered [ 6735.372173] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6739.848821] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6739.859635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6741.098569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6743.239581] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 6748.032296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6748.253487] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6750.340265] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 6754.103919] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6756.996917] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6761.220532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6761.327182] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6761.328820] LustreError: 193444:0:(llog.c:1696:llog_backup()) MGC192.168.202.142@tcp: failed to backup log lustre-OST0000: rc = -28 [ 6761.332828] Lustre: 193444:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.202.142@tcp: can't backup local config lustre-OST0000: rc = -28 [ 6761.385104] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6764.225083] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 6767.565860] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6771.682045] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6771.686893] Lustre: Skipped 1 previous similar message [ 6771.688049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6771.693055] Lustre: Skipped 2 previous similar messages [ 6773.730581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6773.739723] Lustre: Skipped 2 previous similar messages [ 6774.916210] Lustre: server umount lustre-MDT0000 complete [ 6776.701823] LustreError: 191565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786570480 with bad export cookie 2434791087566707802 [ 6776.702148] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6776.706152] LustreError: 191565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6777.823607] LustreError: 192313:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000619227bc x1873354882568704/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6777.912129] Lustre: server umount lustre-MDT0001 complete [ 6790.284213] Lustre: server umount lustre-OST0000 complete [ 6794.688055] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_hostid [ 6799.477355] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 6818.116377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6818.261429] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6818.275479] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6818.327663] Lustre: lustre-MDT0000: new disk, initializing [ 6818.369253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6818.383207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6820.398423] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 6826.373617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6826.444146] Lustre: 196965:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6826.473269] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6826.474781] Lustre: Skipped 1 previous similar message [ 6826.477332] LustreError: 197745:0:(llog.c:1696:llog_backup()) MGC192.168.202.142@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 6826.481749] LustreError: 197745:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 6826.485372] Lustre: 197745:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.142@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 6826.497987] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6826.500502] Lustre: Skipped 1 previous similar message [ 6826.549258] Lustre: lustre-MDT0001: new disk, initializing [ 6826.610543] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6826.635748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6826.643548] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6828.870696] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 6833.897607] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6837.085303] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6840.311675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6840.442106] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6840.443898] Lustre: Skipped 1 previous similar message [ 6840.445494] LustreError: 198907:0:(llog.c:1696:llog_backup()) MGC192.168.202.142@tcp: failed to backup log lustre-OST0000: rc = -28 [ 6840.449922] LustreError: 198907:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 6840.452430] Lustre: 198907:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.142@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 6840.455970] Lustre: 198907:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 6840.464918] Lustre: lustre-OST0000: new disk, initializing [ 6840.467170] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6840.470424] Lustre: 198931:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6840.516534] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6843.179398] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 6847.991164] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6850.092248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6850.101186] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6850.122106] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6850.163208] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6850.235346] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6852.352942] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6852.475430] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6855.137745] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6855.139438] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6855.146459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6855.154897] Lustre: Skipped 1 previous similar message [ 6859.543986] Lustre: server umount lustre-OST0000 complete [ 6862.304290] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6862.310643] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6862.315691] Lustre: Skipped 1 previous similar message [ 6862.317747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6865.376535] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6867.495517] Lustre: server umount lustre-MDT0000 complete [ 6869.306380] LustreError: 197757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786570573 with bad export cookie 2434791087566708495 [ 6869.310738] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6869.312064] LustreError: 197757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6869.461403] Lustre: server umount lustre-MDT0001 complete [ 6873.355903] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 17:36:17 (1786570577) [ 6880.678606] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 6885.470797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6885.666868] LustreError: 202052:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6885.676461] LustreError: 202052:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6885.691925] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6887.623951] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 6890.976611] LustreError: 202053:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6891.020167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6892.808409] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 6893.979394] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6896.923589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6899.864584] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 6903.792546] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6903.900726] Lustre: lustre-OST0001: new disk, initializing [ 6903.903506] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6903.908209] Lustre: 204357:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6906.596290] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6906.610667] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6906.683787] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6906.741079] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 6913.359379] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6919.845498] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6920.699659] Lustre: DEBUG MARKER: ADD OST3 [ 6922.788818] loop: module loaded [ 6922.815886] loop0: detected capacity change from 0 to 204800000 [ 6922.921341] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6922.927078] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6922.933883] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6925.784648] loop0: detected capacity change from 0 to 204800000 [ 6928.491922] Lustre: DEBUG MARKER: STOP OST3 [ 6928.889310] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6928.982153] Lustre: lustre-OST0003: new disk, initializing [ 6928.986943] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 6928.994318] Lustre: 206166:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6929.062404] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 6929.067199] Lustre: Skipped 3 previous similar messages [ 6930.739607] Lustre: *** cfs_fail_loc=2109, val=2*** [ 6930.806966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 6930.809650] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 6930.840776] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 6931.569338] Lustre: Failing over lustre-OST0003 [ 6931.642339] Lustre: server umount lustre-OST0003 complete [ 6932.308996] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 6932.767120] LustreError: 206475:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 6935.083606] LustreError: 203532:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.202.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6936.036802] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6936.038759] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6936.054191] Lustre: Skipped 1 previous similar message [ 6937.182488] Lustre: DEBUG MARKER: START OST3 again [ 6939.781710] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6939.878832] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 6940.186073] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 6941.549657] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 6941.554269] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6942.037060] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 6956.721445] Lustre: Failing over lustre-OST0003 [ 6956.785354] Lustre: server umount lustre-OST0003 complete [ 6957.024491] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6957.028309] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6957.036643] LustreError: 203532:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6957.052311] LustreError: 203532:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6965.735429] LustreError: 203530:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6965.742064] LustreError: 203530:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6970.853207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6970.855907] Lustre: Skipped 5 previous similar messages [ 6975.972230] LustreError: 203533:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6975.979212] LustreError: 203533:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6976.166063] Lustre: server umount lustre-MDT0000 complete [ 6977.863704] LustreError: 205124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786570682 with bad export cookie 2434791087566709377 [ 6977.867543] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6977.869378] LustreError: 205124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6981.088942] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6981.089637] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6981.096623] Lustre: Skipped 6 previous similar messages [ 6981.099293] Lustre: Skipped 4 previous similar messages [ 6992.351116] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6992.474847] Lustre: server umount lustre-MDT0001 complete [ 6994.055675] Lustre: server umount lustre-OST0000 complete [ 6995.458405] Lustre: server umount lustre-OST0001 complete [ 6997.920893] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_hostid [ 7001.084276] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 7014.797678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7014.899108] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7014.913832] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7014.952876] Lustre: lustre-MDT0000: new disk, initializing [ 7014.990968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7014.993965] Lustre: Skipped 1 previous similar message [ 7015.001854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7016.571864] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 7021.219412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7021.276915] Lustre: 211655:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7021.395164] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7022.986739] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 7027.212434] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7029.299645] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7031.948956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7032.068560] Lustre: lustre-OST0000: new disk, initializing [ 7032.070050] Lustre: Skipped 1 previous similar message [ 7032.071904] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7032.074644] Lustre: Skipped 2 previous similar messages [ 7032.077511] Lustre: 213621:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7033.157717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7033.160799] Lustre: Skipped 1 previous similar message [ 7033.167984] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7033.231654] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7034.595787] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 7039.322641] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7041.360118] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7041.491515] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7043.525799] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7043.642706] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7048.673258] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7048.674130] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7048.677084] LustreError: Skipped 2 previous similar messages [ 7048.683178] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7048.686973] Lustre: Skipped 7 previous similar messages [ 7050.452479] Lustre: server umount lustre-OST0000 complete [ 7058.145709] Lustre: server umount lustre-MDT0000 complete [ 7058.912672] LustreError: 212466:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7058.918901] LustreError: 212466:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 7059.594673] LustreError: 211647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786570763 with bad export cookie 2434791087566711659 [ 7059.597973] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7059.601068] LustreError: 211647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7059.749828] Lustre: server umount lustre-MDT0001 complete [ 7061.264064] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 17:39:25 (1786570765) [ 7067.981374] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_hostid [ 7072.146585] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 7086.809932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7086.925285] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7086.944612] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7087.001240] Lustre: lustre-MDT0000: new disk, initializing [ 7087.065600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7088.810420] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 7094.091578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7094.134567] Lustre: 218221:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7094.256656] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7095.945568] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 7100.294704] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7102.477929] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7105.184483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7105.291745] Lustre: 220188:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7107.126769] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7107.716208] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 7112.011135] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7113.725710] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7113.827096] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7115.795074] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7115.898919] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7117.280831] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7117.283829] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7117.284591] LustreError: Skipped 2 previous similar messages [ 7117.287319] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7117.293935] Lustre: Skipped 4 previous similar messages [ 7117.301557] Lustre: Skipped 4 previous similar messages [ 7131.103346] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7131.180119] Lustre: server umount lustre-OST0000 complete [ 7135.200275] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7138.919559] Lustre: server umount lustre-MDT0000 complete [ 7140.511868] LustreError: 218214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786570844 with bad export cookie 2434791087566712541 [ 7140.513889] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7140.520239] LustreError: 218214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7140.672610] Lustre: server umount lustre-MDT0001 complete [ 7143.813579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7144.000583] LustreError: 222403:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7144.031174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7144.033655] Lustre: Skipped 5 previous similar messages [ 7145.559139] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 7148.852675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7150.555914] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 7153.395335] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7155.724075] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7158.597657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7161.078822] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 7163.866074] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7182.068600] Lustre: server umount lustre-OST0000 complete [ 7184.354136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7184.359546] Lustre: Skipped 10 previous similar messages [ 7198.175249] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7198.256299] Lustre: server umount lustre-MDT0000 complete [ 7199.919350] LustreError: 223138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786570904 with bad export cookie 2434791087566713423 [ 7199.921882] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7199.923228] LustreError: 223138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7200.054682] Lustre: server umount lustre-MDT0001 complete [ 7297.937978] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 7299.275521] Key type lgssc unregistered [ 7299.433313] LNet: 226088:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7299.436926] LNetError: 226088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7299.443353] LNet: Removed LNI 192.168.202.142@tcp [ 7299.831152] Key type .llcrypt unregistered [ 7299.833409] Key type ._llcrypt unregistered [ 7310.091860] Key type ._llcrypt registered [ 7310.093133] Key type .llcrypt registered [ 7310.138723] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_hostid [ 7316.992881] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 7317.426993] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7317.443046] alg: No test for adler32 (adler32-zlib) [ 7318.327110] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 7318.432960] LNet: Added LNI 192.168.202.142@tcp [8/256/0/180] [ 7320.031129] Key type lgssc registered [ 7320.492414] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7334.480950] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7334.486338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7335.570491] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7335.582617] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7335.621484] Lustre: lustre-MDT0000: new disk, initializing [ 7335.644562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7335.651799] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7337.195123] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 7341.451746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7341.498755] Lustre: 229414:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7341.522899] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7341.525374] Lustre: Skipped 1 previous similar message [ 7341.561359] Lustre: lustre-MDT0001: new disk, initializing [ 7341.595173] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7341.607028] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7341.611238] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7342.985726] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 7346.799388] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7348.746440] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7351.256922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7351.362039] Lustre: lustre-OST0000: new disk, initializing [ 7351.365051] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7351.369655] Lustre: 231383:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7351.402060] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7353.614932] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 7357.756420] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7359.447572] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7359.483667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7359.487944] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7359.525444] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7360.560071] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 7362.196609] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7362.279950] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7364.577017] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7364.578601] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7364.580034] LustreError: Skipped 1 previous similar message [ 7364.580630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7364.584835] Lustre: Skipped 1 previous similar message [ 7369.114441] Lustre: server umount lustre-OST0000 complete [ 7372.255947] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7372.258326] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7372.262496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7372.264739] Lustre: Skipped 1 previous similar message [ 7374.816123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7374.816658] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7376.662977] Lustre: server umount lustre-MDT0000 complete [ 7377.938128] LustreError: 229406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786571082 with bad export cookie 5587108339424633582 [ 7377.941445] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7377.941745] LustreError: 229406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7378.058789] Lustre: server umount lustre-MDT0001 complete [ 7384.362399] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 7388.181497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7388.336324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7389.878393] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 7393.007914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7393.156664] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7394.633688] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 7395.739963] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7398.395830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7400.830125] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 7404.178185] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7404.259328] Lustre: lustre-OST0001: new disk, initializing [ 7404.261494] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7404.265327] Lustre: 236620:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7404.305315] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7404.307730] Lustre: Skipped 1 previous similar message [ 7406.547536] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 7411.724684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7411.728801] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7411.764741] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7416.302235] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7422.149671] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7426.143263] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 17:45:30 (1786571130) [ 7429.089036] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7429.093167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7429.093405] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7429.094884] Lustre: Skipped 2 previous similar messages [ 7429.098131] Lustre: Skipped 2 previous similar messages [ 7433.178679] Lustre: server umount lustre-MDT0000 complete [ 7434.212303] LustreError: 234311:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7434.218655] LustreError: 234311:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7434.576358] LustreError: 234297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786571138 with bad export cookie 5587108339424634450 [ 7434.579456] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7434.580948] LustreError: 234297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7434.706946] Lustre: server umount lustre-MDT0001 complete [ 7446.533103] Lustre: server umount lustre-OST0000 complete [ 7458.205680] Lustre: server umount lustre-OST0001 complete [ 7460.444561] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_hostid [ 7464.007714] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 7477.573191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7477.673310] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7477.688524] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7477.726688] Lustre: lustre-MDT0000: new disk, initializing [ 7477.755081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7477.762620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7479.284629] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 7483.938108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7483.979568] Lustre: 240558:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7484.092123] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7485.545649] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 7489.641593] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7491.455953] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7493.674365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7493.784698] Lustre: lustre-OST0000: new disk, initializing [ 7493.786754] Lustre: Skipped 1 previous similar message [ 7493.788716] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7493.791209] Lustre: Skipped 2 previous similar messages [ 7493.793672] Lustre: 242524:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7495.292361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7495.296119] Lustre: Skipped 1 previous similar message [ 7495.298610] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7495.336058] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7495.800712] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 7499.490558] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7500.913565] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7500.986472] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7502.352454] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7502.424690] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7505.175271] Lustre: server umount lustre-OST0000 complete [ 7509.983918] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7509.987494] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7509.992876] Lustre: Skipped 1 previous similar message [ 7509.995192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7516.160614] Lustre: server umount lustre-MDT0000 complete [ 7517.567343] LustreError: 240549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786571221 with bad export cookie 5587108339424636571 [ 7517.571617] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7517.575786] LustreError: 240549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7526.936392] Lustre: server umount lustre-MDT0001 complete [ 7530.190359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7530.371879] LustreError: 244740:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7530.409120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7530.412288] Lustre: Skipped 2 previous similar messages [ 7531.987525] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 7534.923731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7536.492450] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 7539.023190] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7540.813747] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7543.068385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7545.126357] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 7547.489950] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7693.794807] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7693.794862] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7693.796261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7693.796267] Lustre: Skipped 4 previous similar messages [ 7693.798203] LustreError: Skipped 1 previous similar message [ 7693.803579] Lustre: Skipped 2 previous similar messages [ 7698.319274] Lustre: server umount lustre-OST0000 complete [ 7704.044969] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7705.843051] Lustre: server umount lustre-MDT0000 complete [ 7708.392705] LustreError: 247356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786571412 with bad export cookie 5587108339424637453 [ 7708.396357] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7708.401019] LustreError: 247356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7708.594869] Lustre: server umount lustre-MDT0001 complete [ 7819.194841] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 7820.712024] Key type lgssc unregistered [ 7820.906509] LNet: 248348:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7820.909506] LNetError: 248348:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7820.920565] LNet: Removed LNI 192.168.202.142@tcp [ 7821.359339] Key type .llcrypt unregistered [ 7821.363289] Key type ._llcrypt unregistered [ 7833.595526] Key type ._llcrypt registered [ 7833.597032] Key type .llcrypt registered [ 7833.642322] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_hostid [ 7841.422426] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 7841.954369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7841.968700] alg: No test for adler32 (adler32-zlib) [ 7842.874224] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 7843.027423] LNet: Added LNI 192.168.202.142@tcp [8/256/0/180] [ 7844.623162] Key type lgssc registered [ 7845.232052] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7867.325948] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 7873.084771] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7873.093860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7874.250098] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7874.269492] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7874.317571] Lustre: lustre-MDT0000: new disk, initializing [ 7874.376750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7874.385435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7876.652657] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 7882.863942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7882.931411] Lustre: 252389:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7882.970806] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7882.974897] Lustre: Skipped 1 previous similar message [ 7883.019413] Lustre: lustre-MDT0001: new disk, initializing [ 7883.093960] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7883.132049] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7883.145431] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7885.629574] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 7888.750680] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7892.570067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7892.794492] Lustre: lustre-OST0000: new disk, initializing [ 7892.797247] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7892.801876] Lustre: 254024:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7892.856171] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7896.349573] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 7900.700417] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7900.706843] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7900.755437] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7902.919923] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7903.056687] Lustre: lustre-OST0001: new disk, initializing [ 7903.058766] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7903.062248] Lustre: 254894:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7903.122120] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7906.970148] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 7910.436986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7910.444663] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7910.468062] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7915.132996] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7926.159512] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7934.433504] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7934.437604] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7934.442429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7935.972271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7935.976366] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7935.983653] Lustre: Skipped 2 previous similar messages [ 7941.088704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7941.091791] Lustre: Skipped 6 previous similar messages [ 7946.207194] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7946.210950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7946.213962] Lustre: Skipped 3 previous similar messages [ 7946.308476] Lustre: server umount lustre-MDT0000 complete [ 7947.854376] LustreError: 252381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786571652 with bad export cookie 12594518535066546630 [ 7947.857712] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7947.859638] LustreError: 252381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7947.993143] Lustre: server umount lustre-MDT0001 complete [ 7959.576567] Lustre: server umount lustre-OST0000 complete [ 7971.799262] Lustre: server umount lustre-OST0001 complete [ 7977.354470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7981.071344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7984.379650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7987.831547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7995.313353] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 7998.987995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7998.999383] Lustre: lustre-MDT0000: reset Object Index mappings [ 7999.279105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8000.947250] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 8004.577114] LustreError: 259892:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8004.586628] LustreError: 259892:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8004.602865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8004.614113] Lustre: lustre-MDT0001: reset Object Index mappings [ 8006.613241] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 8007.725048] Lustre: 261031:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8010.322870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8010.501762] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8010.504877] Lustre: Skipped 1 previous similar message [ 8013.056600] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 8014.626270] LustreError: 261386:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8016.646512] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8019.596957] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 8024.046592] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 8024.047626] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8028.874545] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8030.389498] Lustre: 262903:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8034.273997] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8034.276530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8034.278782] Lustre: Skipped 2 previous similar messages [ 8034.281086] Lustre: Skipped 1 previous similar message [ 8037.976543] Lustre: server umount lustre-MDT0000 complete [ 8039.394284] LustreError: 259892:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8039.401937] LustreError: 259892:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8039.420695] LustreError: 259873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786571743 with bad export cookie 12594518535066553420 [ 8039.421398] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8039.426296] LustreError: 259873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8039.569798] Lustre: server umount lustre-MDT0001 complete [ 8051.719659] Lustre: server umount lustre-OST0000 complete [ 8063.705123] Lustre: server umount lustre-OST0001 complete [ 8071.781683] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 17:56:15 (1786571775) [ 8078.966611] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_hostid [ 8083.085746] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 8098.543893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8098.660359] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8098.675806] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8098.722299] Lustre: lustre-MDT0000: new disk, initializing [ 8098.764508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8098.766992] Lustre: Skipped 1 previous similar message [ 8098.774790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8100.721139] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 8105.440615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8105.488930] Lustre: 267151:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8105.618246] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8107.264654] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 8111.252544] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8113.183542] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8115.805338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8115.936586] Lustre: lustre-OST0000: new disk, initializing [ 8115.938454] Lustre: Skipped 1 previous similar message [ 8115.940754] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8115.943596] Lustre: Skipped 2 previous similar messages [ 8115.946038] Lustre: 269116:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8117.055844] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8117.058637] Lustre: Skipped 1 previous similar message [ 8117.060876] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8117.097301] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8118.277945] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 8122.357135] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8124.107393] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8124.207355] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8125.824496] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8125.923062] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8127.456715] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8127.457983] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8127.460109] LustreError: Skipped 1 previous similar message [ 8127.465289] Lustre: Skipped 1 previous similar message [ 8127.467745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8127.470893] Lustre: Skipped 3 previous similar messages [ 8132.835078] Lustre: server umount lustre-OST0000 complete [ 8136.674045] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8136.676927] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8140.375279] Lustre: server umount lustre-MDT0000 complete [ 8141.707644] LustreError: 267143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786571845 with bad export cookie 12594518535066555492 [ 8141.708662] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8141.721084] LustreError: 267143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8141.842211] Lustre: server umount lustre-MDT0001 complete [ 8148.573989] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 8152.238215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8152.376356] LustreError: 272047:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8152.389609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8152.392038] Lustre: Skipped 2 previous similar messages [ 8153.748665] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 8156.681622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8158.234244] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 8159.248963] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8161.526685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8163.670633] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 8166.762789] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8166.821993] Lustre: lustre-OST0001: new disk, initializing [ 8166.823592] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8166.826487] Lustre: 274351:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8168.709627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8168.713678] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8168.746319] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8168.977849] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 8173.480915] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8174.740986] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8176.741258] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 8176.751714] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 8179.175551] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8179.178579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8179.191243] Lustre: Skipped 3 previous similar messages [ 8179.193727] Lustre: Skipped 6 previous similar messages [ 8184.742853] Lustre: server umount lustre-MDT0000 complete [ 8186.107437] LustreError: 274350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786571890 with bad export cookie 12594518535066556374 [ 8186.112080] LustreError: 274350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8186.112449] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8186.244213] Lustre: server umount lustre-MDT0001 complete [ 8197.636621] Lustre: server umount lustre-OST0000 complete [ 8209.378858] Lustre: server umount lustre-OST0001 complete [ 8214.999240] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 8218.701075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8218.832623] LustreError: 277404:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8218.852461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8218.856187] Lustre: Skipped 3 previous similar messages [ 8220.331896] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 8223.230021] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8224.877473] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 8225.938368] Lustre: 278545:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8228.388664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8229.538676] LustreError: 278900:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8229.544449] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 8230.459649] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 8233.364463] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8234.478898] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 8235.441979] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 8238.817247] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8240.083326] Lustre: 280419:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 8241.896099] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 8241.897942] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 8245.146085] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 17:59:09 (1786571949) [ 8259.551857] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8259.555086] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8259.560513] Lustre: Skipped 1 previous similar message [ 8259.563568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8259.566161] Lustre: Skipped 6 previous similar messages [ 8262.875804] Lustre: server umount lustre-MDT0000 complete [ 8264.143256] LustreError: 278548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786571968 with bad export cookie 12594518535066562674 [ 8264.147032] LustreError: 278548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8264.147334] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8265.183985] LustreError: 278921:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8265.192869] LustreError: 278921:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8270.335440] Lustre: server umount lustre-MDT0001 complete [ 8286.175145] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8286.213773] Lustre: server umount lustre-OST0000 complete [ 8293.594019] Lustre: server umount lustre-OST0001 complete [ 8295.696652] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_hostid [ 8298.895160] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 8314.645496] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 8318.022191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8318.115390] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8318.126726] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8318.168323] Lustre: lustre-MDT0000: new disk, initializing [ 8318.202826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8319.734553] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 8324.188980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8324.255943] Lustre: 284651:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8324.405123] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8325.990216] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 8331.261608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8331.375363] Lustre: 286283:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8333.423090] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8333.658696] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 8338.293970] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8339.512147] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8340.639512] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 8345.284119] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8350.663162] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 18:00:54 (1786572054) [ 8354.787934] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8354.789993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8354.793317] Lustre: Skipped 8 previous similar messages [ 8354.794785] Lustre: Skipped 10 previous similar messages [ 8357.467789] Lustre: server umount lustre-MDT0000 complete [ 8358.651164] LustreError: 284645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786572062 with bad export cookie 12594518535066570528 [ 8358.655171] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8358.655575] LustreError: 284645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8358.779060] Lustre: server umount lustre-MDT0001 complete [ 8370.375722] Lustre: server umount lustre-OST0000 complete [ 8380.901177] Lustre: server umount lustre-OST0001 complete [ 8384.047198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8384.240284] LustreError: 289408:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8384.246677] LustreError: 289408:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8384.278744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8384.282032] Lustre: Skipped 7 previous similar messages [ 8385.935605] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 8389.202795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8391.046976] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 8394.094458] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8396.248656] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8398.833796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8401.255472] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 8403.831079] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8420.157018] loop1: detected capacity change from 0 to 204800000 [ 8420.257312] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8420.261876] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8420.269453] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8421.948227] loop1: detected capacity change from 0 to 204800000 [ 8423.802112] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8423.835657] LustreError: 289396:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs23668: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 8423.841258] LustreError: 289396:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs23668 [ 8423.843686] LustreError: 289396:0:(mgs_handler.c:517:mgs_target_reg()) fs23668-MDT0001: Error checking target index: rc = -13 [ 8423.847688] LustreError: fs23668-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 8423.850588] LustreError: 293144:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 8423.853534] LustreError: 293144:0:(tgt_mount.c:2129:server_put_super()) no obd fs23668-MDT0001 [ 8423.856225] LustreError: 293144:0:(tgt_mount.c:116:server_deregister_mount()) fs23668-MDT0001 not registered [ 8423.866725] Lustre: server umount fs23668-MDT0001 complete [ 8423.868526] LustreError: 293144:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 8426.722984] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8426.752956] Lustre: Setting parameter fs23668-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs23668-MDT0001 [ 8426.757072] Lustre: Skipped 2 previous similar messages [ 8426.773596] Lustre: srv-fs23668-MDT0001: No data found on store. Initialize space. [ 8426.775454] Lustre: Skipped 4 previous similar messages [ 8426.808717] Lustre: fs23668-MDT0001: new disk, initializing [ 8426.810224] Lustre: Skipped 3 previous similar messages [ 8426.843452] Lustre: srv-fs23668-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 8426.846766] Lustre: 293611:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs23668-MDT0001-mdtlov: can't create new llog: rc = -115 [ 8426.850613] LustreError: 293611:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs23668-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 8428.204739] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 8431.113918] Lustre: server umount fs23668-MDT0001 complete [ 8435.362075] loop1: detected capacity change from 0 to 204800000 [ 8437.232627] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8437.338321] Lustre: 295176:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs23668-MDT0001-mdtlov: can't create new llog: rc = -115 [ 8437.341620] LustreError: 295176:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs23668-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 8438.712647] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 8440.332680] Lustre: server umount fs23668-MDT0001 complete [ 8445.408610] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8445.412660] Lustre: Skipped 2 previous similar messages [ 8449.628575] Lustre: server umount lustre-MDT0000 complete [ 8450.072393] LustreError: 290712:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8450.079361] LustreError: 290712:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 25 previous similar messages [ 8452.653070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8452.699383] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8454.184958] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 8456.883085] loop1: detected capacity change from 0 to 204800000 [ 8458.214266] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8458.897799] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8459.044526] Lustre: 297619:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs23668-MDT0001-mdtlov: can't create new llog: rc = -115 [ 8459.049254] LustreError: 297619:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs23668-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 8460.565269] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 8462.315275] Lustre: server umount fs23668-MDT0001 complete [ 8472.031851] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8475.793461] Lustre: server umount lustre-OST0000 complete [ 8483.297052] Lustre: server umount lustre-MDT0000 complete [ 8484.543114] LustreError: 294154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786572188 with bad export cookie 12594518535066573202 [ 8484.547854] LustreError: 294154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8484.669836] Lustre: server umount lustre-MDT0001 complete [ 8488.395404] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 8489.487331] Key type lgssc unregistered [ 8489.621442] LNet: 299962:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8489.625109] LNetError: 299962:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8489.633473] LNet: Removed LNI 192.168.202.142@tcp [ 8489.970117] Key type .llcrypt unregistered [ 8489.971761] Key type ._llcrypt unregistered [ 8497.694472] Key type ._llcrypt registered [ 8497.695727] Key type .llcrypt registered [ 8497.732597] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 8498.055365] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8498.094196] alg: No test for adler32 (adler32-zlib) [ 8498.956408] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 8499.052992] LNet: Added LNI 192.168.202.142@tcp [8/256/0/180] [ 8500.639153] Key type lgssc registered [ 8501.031967] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8503.741523] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8503.747415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8504.904930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8506.279763] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 8509.032140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8509.170819] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8510.519511] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 8512.868208] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8514.669753] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8516.884439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8516.999660] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8518.886899] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 8519.075493] LustreError: 303037:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8519.081391] LustreError: 303037:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8521.219710] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8524.257132] LustreError: 303039:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8529.377033] LustreError: 303039:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8532.004783] LustreError: 303038:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8532.012383] LustreError: 303038:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8537.118506] LustreError: 303037:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8537.125803] LustreError: 303037:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8539.960581] loop1: detected capacity change from 0 to 204800000 [ 8540.071292] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8540.085419] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8540.098415] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8543.203153] loop1: detected capacity change from 0 to 204800000 [ 8545.770499] LustreError: 303038:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8545.799389] LustreError: 303038:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 8547.034597] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8547.153652] Lustre: fs19515-OST0000: new disk, initializing [ 8547.157300] Lustre: srv-fs19515-OST0000: No data found on store. Initialize space. [ 8547.162032] Lustre: 304961:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8547.231390] Lustre: fs19515-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8551.323737] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 8557.181214] Lustre: server umount fs19515-OST0000 complete [ 8562.725683] LustreError: 303038:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8562.733784] LustreError: 303038:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 8566.072693] loop1: detected capacity change from 0 to 204800000 [ 8570.398438] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8570.555778] Lustre: fs19515-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8574.407173] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 8579.144360] loop2: detected capacity change from 0 to 204800000 [ 8579.268668] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8579.278535] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8579.284474] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8582.104252] loop2: detected capacity change from 0 to 204800000 [ 8585.383584] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8585.446242] LustreError: fs19515-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 8585.453491] LustreError: fs19515-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 8585.461456] LustreError: 307998:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 8585.466658] LustreError: 307998:0:(tgt_mount.c:2129:server_put_super()) no obd fs19515-OST0001 [ 8585.470450] LustreError: 307998:0:(tgt_mount.c:116:server_deregister_mount()) fs19515-OST0001 not registered [ 8585.493732] Lustre: server umount fs19515-OST0001 complete [ 8585.497402] LustreError: 307998:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 8592.013637] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8592.096696] Lustre: fs19515-OST0001: new disk, initializing [ 8592.100954] Lustre: srv-fs19515-OST0001: No data found on store. Initialize space. [ 8592.106972] Lustre: 308549:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8592.160906] Lustre: fs19515-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8594.922024] LustreError: 303426:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8594.930675] LustreError: 303426:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [ 8596.118893] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 8602.268712] Lustre: server umount fs19515-OST0000 complete [ 8605.157111] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [ 8607.393462] Lustre: server umount fs19515-OST0001 complete [ 8614.113570] loop1: detected capacity change from 0 to 204800000 [ 8615.409728] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [ 8616.987542] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8617.131522] Lustre: fs19515-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8620.000911] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 8623.720190] loop2: detected capacity change from 0 to 204800000 [ 8626.916787] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8630.218159] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 8634.712151] Lustre: server umount fs19515-OST0000 complete [ 8639.583534] Lustre: server umount fs19515-OST0001 complete [ 8646.120263] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8646.123721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8646.130148] Lustre: Skipped 1 previous similar message [ 8646.135658] Lustre: Skipped 1 previous similar message [ 8651.233427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8651.839365] Lustre: server umount lustre-OST0000 complete [ 8656.352837] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8656.353989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8656.363111] Lustre: Skipped 1 previous similar message [ 8656.365814] Lustre: Skipped 1 previous similar message [ 8661.481117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8661.484982] Lustre: Skipped 2 previous similar messages [ 8666.592144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8666.594589] Lustre: Skipped 1 previous similar message [ 8668.127146] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8668.203836] Lustre: server umount lustre-MDT0000 complete [ 8669.970675] LustreError: 301191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786572374 with bad export cookie 4842984106416317434 [ 8669.974307] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8669.975151] LustreError: 301191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8670.128785] Lustre: server umount lustre-MDT0001 complete [ 8676.113776] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 8677.434667] Key type lgssc unregistered [ 8677.596327] LNet: 314471:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8677.602289] LNetError: 314471:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8677.618072] LNet: Removed LNI 192.168.202.142@tcp [ 8678.030740] Key type .llcrypt unregistered [ 8678.033787] Key type ._llcrypt unregistered [ 8693.585635] Key type ._llcrypt registered [ 8693.587349] Key type .llcrypt registered [ 8693.641352] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_hostid [ 8702.061926] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 8702.531992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8702.594080] alg: No test for adler32 (adler32-zlib) [ 8703.481874] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 8703.605176] LNet: Added LNI 192.168.202.142@tcp [8/256/0/180] [ 8705.239158] Key type lgssc registered [ 8705.871454] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8723.023423] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8723.031707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8724.121825] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8724.160488] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8724.200891] Lustre: lustre-MDT0000: new disk, initializing [ 8724.232156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8724.240369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8725.896673] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 8730.720213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8730.776543] Lustre: 318302:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8730.821041] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8730.823493] Lustre: Skipped 1 previous similar message [ 8730.860797] Lustre: lustre-MDT0001: new disk, initializing [ 8730.894506] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8730.911376] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8730.916743] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8732.426316] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 8736.773587] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8739.269931] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8742.308150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8742.433992] Lustre: lustre-OST0000: new disk, initializing [ 8742.436680] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8742.440171] Lustre: 320270:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8742.480498] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8744.846332] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 8748.998381] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8750.408311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8750.412249] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8750.459935] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8755.700284] Lustre: Mounted lustre-client - version 2.17.57_2_g9cc1fed [ 8756.123993] Lustre: Unmounted lustre-client [ 8760.802041] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8760.802578] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8760.805380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8760.806301] Lustre: Skipped 1 previous similar message [ 8760.808957] LustreError: Skipped 1 previous similar message [ 8765.915883] Lustre: server umount lustre-OST0000 complete [ 8771.040506] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8771.041252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8771.045807] Lustre: Skipped 1 previous similar message [ 8771.048925] Lustre: Skipped 1 previous similar message [ 8773.670665] Lustre: server umount lustre-MDT0000 complete [ 8775.093420] LustreError: 318295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786572479 with bad export cookie 12052615031589706125 [ 8775.096111] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8775.097175] LustreError: 318295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8775.217174] Lustre: server umount lustre-MDT0001 complete [ 8779.165147] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 8780.366489] Key type lgssc unregistered [ 8808.095168] LNet: 1 local NIs in recovery (showing 1): 192.168.202.142@tcp [ 8808.159382] LNet: 322432:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8808.162160] LNetError: 322432:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8881.893443] LNet: Removed LNI 192.168.202.142@tcp [ 8882.236534] Key type .llcrypt unregistered [ 8882.237913] Key type ._llcrypt unregistered [ 8889.646988] Key type ._llcrypt registered [ 8889.649297] Key type .llcrypt registered [ 8889.697081] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_hostid [ 8897.167725] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 8897.692461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8897.711577] alg: No test for adler32 (adler32-zlib) [ 8898.606793] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 8898.721703] LNet: Added LNI 192.168.202.142@tcp [8/256/0/180] [ 8900.319167] Key type lgssc registered [ 8900.746372] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8914.438472] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 18:10:18 (1786572618) [ 8917.346382] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8917.353717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8918.472092] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8918.494673] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8918.536149] Lustre: lustre-MDT0000: new disk, initializing [ 8918.570223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8918.583948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8920.140943] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 8924.665961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8924.714201] Lustre: 325630:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8924.742443] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8924.745814] Lustre: Skipped 1 previous similar message [ 8924.780828] Lustre: lustre-MDT0001: new disk, initializing [ 8924.813907] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8924.826209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8924.829274] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8926.265115] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 8930.386646] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8932.623838] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8935.509255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8935.650950] Lustre: lustre-OST0000: new disk, initializing [ 8935.653383] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8935.660803] Lustre: 327597:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8935.701697] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8938.117928] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 8941.051538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8941.055859] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8941.090564] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8942.427705] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8951.264800] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8951.265741] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8951.267716] LustreError: Skipped 1 previous similar message [ 8951.269214] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8951.272514] Lustre: Skipped 1 previous similar message [ 8954.777853] Lustre: server umount lustre-OST0000 complete [ 8956.384423] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8956.385147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8956.389129] Lustre: Skipped 1 previous similar message [ 8956.393088] Lustre: Skipped 2 previous similar messages [ 8961.504285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8961.507454] Lustre: Skipped 1 previous similar message [ 8962.554554] Lustre: server umount lustre-MDT0000 complete [ 8964.191227] LustreError: 325620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786572668 with bad export cookie 3682117266710503198 [ 8964.194563] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8964.195540] LustreError: 325620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8964.315946] Lustre: server umount lustre-MDT0001 complete [ 8968.360684] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 8969.638206] Key type lgssc unregistered [ 8969.787532] LNet: 329503:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8969.789933] LNetError: 329503:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8969.802149] LNet: Removed LNI 192.168.202.142@tcp [ 8970.165875] Key type .llcrypt unregistered [ 8970.167171] Key type ._llcrypt unregistered [ 8979.158737] Key type ._llcrypt registered [ 8979.160370] Key type .llcrypt registered [ 8979.201259] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 8979.667906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8979.675858] alg: No test for adler32 (adler32-zlib) [ 8980.556092] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 8980.670289] LNet: Added LNI 192.168.202.142@tcp [8/256/0/180] [ 8982.255124] Key type lgssc registered [ 8982.688322] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8985.441487] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8985.451297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8986.632349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8988.046653] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 8991.069086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8991.211129] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8992.682613] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 8995.057911] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8997.035950] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8999.421566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8999.545779] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9001.635153] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 9004.228027] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9018.264532] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9018.855201] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9021.919847] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9021.923209] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9021.928791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9022.434371] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9022.435395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9022.441318] Lustre: Skipped 2 previous similar messages [ 9025.057048] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.42@tcp (stopping) [ 9026.326851] Lustre: server umount lustre-MDT0000 complete [ 9027.552605] LustreError: 330755:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9027.557878] LustreError: 330755:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 9027.709260] LustreError: 330737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786572731 with bad export cookie 5955674275937696906 [ 9027.712462] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9027.714535] LustreError: 330737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9030.168238] LustreError: 330751:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9030.176352] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.42@tcp (stopping) [ 9032.671893] LustreError: 330750:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9032.672174] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9032.682601] Lustre: Skipped 1 previous similar message [ 9035.288968] LustreError: 330749:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9035.291668] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.42@tcp (stopping) [ 9035.299157] Lustre: Skipped 1 previous similar message [ 9040.409228] LustreError: 330750:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9040.418351] LustreError: 330750:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9041.890831] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9041.989825] Lustre: server umount lustre-MDT0001 complete [ 9049.554581] Lustre: server umount lustre-OST0000 complete [ 9052.531170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9053.961874] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 9054.989940] Lustre: 334950:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.142@tcp [ 9055.761817] Lustre: server umount MGS complete [ 9058.839359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9058.991549] LustreError: 335521:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9059.020915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9060.494415] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 9063.576721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9063.712528] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9065.432051] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 9068.972191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9071.569303] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 9074.682060] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9075.335266] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [ 9077.513141] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9087.233052] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 9 sec [ 9088.479849] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9088.482396] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9088.487414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9088.490606] Lustre: Skipped 2 previous similar messages [ 9089.504960] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9090.453541] Lustre: server umount lustre-OST0000 complete [ 9093.846785] Lustre: server umount lustre-MDT0000 complete [ 9094.624277] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9094.624907] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9094.631276] Lustre: Skipped 1 previous similar message [ 9094.633566] LustreError: 335522:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9094.638140] LustreError: 335522:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9095.191386] LustreError: 337917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786572799 with bad export cookie 5955674275937697732 [ 9095.194049] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9095.197719] LustreError: 337917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9095.323268] Lustre: server umount lustre-MDT0001 complete [ 9099.280492] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 9100.370369] Key type lgssc unregistered [ 9100.493513] LNet: 338959:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9100.496302] LNetError: 338959:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9100.504324] LNet: Removed LNI 192.168.202.142@tcp [ 9100.822490] Key type .llcrypt unregistered [ 9100.824057] Key type ._llcrypt unregistered [ 9107.399266] Key type ._llcrypt registered [ 9107.400651] Key type .llcrypt registered [ 9107.438893] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_hostid [ 9114.285389] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 9114.646528] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9114.701659] alg: No test for adler32 (adler32-zlib) [ 9115.561904] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 9115.654607] LNet: Added LNI 192.168.202.142@tcp [8/256/0/180] [ 9117.231169] Key type lgssc registered [ 9117.646206] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9130.843191] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9130.850137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9131.942578] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9131.957465] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9131.997332] Lustre: lustre-MDT0000: new disk, initializing [ 9132.042207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9132.052053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9133.592364] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 9138.193698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9138.240314] Lustre: 342093:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9138.266975] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9138.269326] Lustre: Skipped 1 previous similar message [ 9138.304925] Lustre: lustre-MDT0001: new disk, initializing [ 9138.341561] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9138.353883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9138.357339] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9139.882232] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 9143.989198] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9146.109931] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9148.486894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9148.596118] Lustre: lustre-OST0000: new disk, initializing [ 9148.598094] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9148.601632] Lustre: 344059:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9148.632885] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9148.753527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9148.757320] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9148.795199] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9150.681891] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 9154.567586] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9156.186477] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9156.270267] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9157.743847] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9157.831310] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9159.137292] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9159.137914] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9159.141244] LustreError: Skipped 1 previous similar message [ 9159.147365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9159.151277] Lustre: Skipped 1 previous similar message [ 9164.258264] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9164.261381] Lustre: Skipped 1 previous similar message [ 9164.631825] Lustre: server umount lustre-OST0000 complete [ 9169.377835] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9169.377857] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9169.378887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9169.385677] Lustre: Skipped 2 previous similar messages [ 9172.195167] Lustre: server umount lustre-MDT0000 complete [ 9173.601504] LustreError: 342084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786572877 with bad export cookie 16624073019793997954 [ 9173.605255] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9173.605511] LustreError: 342084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9173.735153] Lustre: server umount lustre-MDT0001 complete [ 9177.065840] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 18:14:41 (1786572881) [ 9182.905609] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_hostid [ 9186.626121] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 9199.281421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9199.362510] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9199.373296] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9199.409434] Lustre: lustre-MDT0000: new disk, initializing [ 9199.430164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9199.436660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9200.852714] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 9205.292320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9205.334606] Lustre: 348808:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9205.356590] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9205.358350] Lustre: Skipped 1 previous similar message [ 9205.393616] Lustre: lustre-MDT0001: new disk, initializing [ 9205.427562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9205.439976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9205.444077] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9206.851862] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 9210.714352] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9212.692446] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9215.287418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9215.394055] Lustre: lustre-OST0000: new disk, initializing [ 9215.396030] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9215.399250] Lustre: 350778:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9215.428794] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9215.629887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9215.633788] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9215.672074] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9217.522209] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 9221.413125] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9231.328874] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9231.331528] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9231.332413] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9231.336774] Lustre: Skipped 1 previous similar message [ 9231.339749] Lustre: Skipped 1 previous similar message [ 9236.450056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9236.452660] Lustre: Skipped 2 previous similar messages [ 9236.947286] Lustre: server umount lustre-OST0000 complete [ 9241.568828] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9241.568884] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9241.575069] LustreError: Skipped 1 previous similar message [ 9244.453387] Lustre: server umount lustre-MDT0000 complete [ 9245.780374] LustreError: 348799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786572949 with bad export cookie 16624073019793998822 [ 9245.782534] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9245.788033] LustreError: 348799:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9245.915145] Lustre: server umount lustre-MDT0001 complete [ 9249.869809] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 9251.054460] Key type lgssc unregistered [ 9251.181468] LNet: 352928:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9251.184588] LNetError: 352928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9251.190336] LNet: Removed LNI 192.168.202.142@tcp [ 9251.514232] Key type .llcrypt unregistered [ 9251.515820] Key type ._llcrypt unregistered [ 9284.841092] Key type ._llcrypt registered [ 9284.842415] Key type .llcrypt registered [ 9284.884358] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 9285.001211] Key type .llcrypt unregistered [ 9285.002615] Key type ._llcrypt unregistered [ 9313.694594] Key type ._llcrypt registered [ 9313.695582] Key type .llcrypt registered [ 9313.735126] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 9313.859439] Key type .llcrypt unregistered [ 9313.860867] Key type ._llcrypt unregistered [ 9325.168744] Key type ._llcrypt registered [ 9325.170333] Key type .llcrypt registered [ 9325.209835] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 9325.322410] Key type .llcrypt unregistered [ 9325.324523] Key type ._llcrypt unregistered [ 9336.543653] Key type ._llcrypt registered [ 9336.544852] Key type .llcrypt registered [ 9336.584201] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 9336.700196] Key type .llcrypt unregistered [ 9336.701407] Key type ._llcrypt unregistered [ 9368.703523] Key type ._llcrypt registered [ 9368.704850] Key type .llcrypt registered [ 9368.747909] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 9368.891414] Key type .llcrypt unregistered [ 9368.894231] Key type ._llcrypt unregistered [ 9380.105752] Key type ._llcrypt registered [ 9380.107308] Key type .llcrypt registered [ 9380.155653] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 9380.268489] Key type .llcrypt unregistered [ 9380.270066] Key type ._llcrypt unregistered [ 9391.433652] Key type ._llcrypt registered [ 9391.434591] Key type .llcrypt registered [ 9391.471626] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 9391.576281] Key type .llcrypt unregistered [ 9391.577314] Key type ._llcrypt unregistered [ 9402.455834] Key type ._llcrypt registered [ 9402.457098] Key type .llcrypt registered [ 9402.491643] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 9402.581320] Key type .llcrypt unregistered [ 9402.582242] Key type ._llcrypt unregistered [ 9413.413337] Key type ._llcrypt registered [ 9413.414666] Key type .llcrypt registered [ 9413.454020] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 9413.572527] Key type .llcrypt unregistered [ 9413.574783] Key type ._llcrypt unregistered [ 9424.171758] Key type ._llcrypt registered [ 9424.172888] Key type .llcrypt registered [ 9424.209161] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 9424.312301] Key type .llcrypt unregistered [ 9424.313704] Key type ._llcrypt unregistered [ 9445.205636] Key type ._llcrypt registered [ 9445.206859] Key type .llcrypt registered [ 9445.245741] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 9445.364533] Key type .llcrypt unregistered [ 9445.365589] Key type ._llcrypt unregistered [ 9456.189295] Key type ._llcrypt registered [ 9456.190376] Key type .llcrypt registered [ 9456.225302] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 9456.331470] Key type .llcrypt unregistered [ 9456.332925] Key type ._llcrypt unregistered [ 9467.479148] Key type ._llcrypt registered [ 9467.480541] Key type .llcrypt registered [ 9467.518154] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 9467.630425] Key type .llcrypt unregistered [ 9467.631832] Key type ._llcrypt unregistered [ 9486.562357] Key type ._llcrypt registered [ 9486.564105] Key type .llcrypt registered [ 9486.602922] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_hostid [ 9493.186023] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 9493.490602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9493.514474] alg: No test for adler32 (adler32-zlib) [ 9494.361349] Lustre: Lustre: Build Version: 2.17.57_2_g9cc1fed [ 9494.445385] LNet: Added LNI 192.168.202.142@tcp [8/256/0/180] [ 9496.023148] Key type lgssc registered [ 9496.453766] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9507.987643] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9507.994050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9509.069204] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 9509.080229] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 9509.120380] Lustre: tst250fs-MDT0000: new disk, initializing [ 9509.140353] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9509.147318] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9510.448475] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 9514.509815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9514.548165] Lustre: 372898:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9514.569494] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 9514.571187] Lustre: Skipped 1 previous similar message [ 9514.603522] Lustre: tst250fs-MDT0001: new disk, initializing [ 9514.635573] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9514.646973] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9514.651065] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9515.950963] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 9518.205943] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9520.376504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9520.476802] Lustre: tst250fs-OST0000: new disk, initializing [ 9520.479108] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 9520.482513] Lustre: 374529:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9520.515944] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9521.973855] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9521.977887] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9522.017921] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9522.451217] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 9526.901554] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9526.987205] Lustre: tst250fs-OST0001: new disk, initializing [ 9526.989603] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 9526.993331] Lustre: 375401:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9527.044742] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9529.263918] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_default_debug -1 all [ 9530.123142] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9530.128972] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9530.159638] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9532.074284] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 9545.699325] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9545.701473] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9545.702258] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 9545.703646] Lustre: Skipped 3 previous similar messages [ 9548.767084] Lustre: server umount tst250fs-MDT0000 complete [ 9550.129324] LustreError: 374528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786573254 with bad export cookie 2467693058790538523 [ 9550.131984] LustreError: MGC192.168.202.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9550.134111] LustreError: 374528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9550.265134] Lustre: server umount tst250fs-MDT0001 complete [ 9560.835596] Lustre: server umount tst250fs-OST0000 complete [ 9572.322557] Lustre: server umount tst250fs-OST0001 complete [ 9578.145662] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_hostid [ 9580.628661] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 9590.941368] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 18:21:34 (1786573294) [ 9591.544691] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 9597.774499] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing set_hostid [ 9601.341753] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing load_modules_local [ 9612.991304] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 9265 sec ========== 18:21:56 (1786573316) [ 9613.605662] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 18:21:57 (1786573317) === [ 9614.752299] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 18:21:58 (1786573318) === [ 9628.876433] Lustre: DEBUG MARKER: oleg242-server.virtnet: executing unload_modules_local [ 9629.914131] Key type lgssc unregistered [ 9630.037290] LNet: 384513:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9630.039922] LNetError: 384513:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9630.048436] LNet: Removed LNI 192.168.202.142@tcp [ 9630.366129] Key type .llcrypt unregistered [ 9630.367555] Key type ._llcrypt unregistered