[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 3.0.0 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 504334293 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f53f0-0x000f53ff] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5200 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1D87 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1C23 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001BE3 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1C97 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1D27 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1D5F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1c23-0xbffe1c96] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1c22] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1c97-0xbffe1d26] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1d27-0xbffe1d5e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1d5f-0xbffe1d86] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002000] x2apic enabled [ 0.002000] Switched APIC routing to physical x2apic. [ 0.002015] kvm-guest: setup PV IPIs [ 0.005000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.005000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.005033] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.006017] pid_max: default: 32768 minimum: 301 [ 0.007165] LSM: Security Framework initializing [ 0.008087] Yama: becoming mindful. [ 0.009045] SELinux: Initializing. [ 0.010099] *** VALIDATE selinux *** [ 0.019649] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024481] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025167] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026144] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027103] *** VALIDATE tmpfs *** [ 0.029166] *** VALIDATE proc *** [ 0.030212] *** VALIDATE cgroup *** [ 0.031008] *** VALIDATE cgroup2 *** [ 0.033203] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034154] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036024] Spectre V2 : User space: Vulnerable [ 0.037011] Speculative Store Bypass: Vulnerable [ 0.040141] debug: unmapping init [mem 0xffffffff91a59000-0xffffffff91a60fff] [ 0.043173] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044686] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045024] ... version: 2 [ 0.046012] ... bit width: 48 [ 0.047010] ... generic registers: 4 [ 0.048010] ... value mask: 0000ffffffffffff [ 0.049013] ... max period: 00007fffffffffff [ 0.050013] ... fixed-purpose events: 3 [ 0.051010] ... event mask: 000000070000000f [ 0.052324] rcu: Hierarchical SRCU implementation. [ 0.054310] smp: Bringing up secondary CPUs ... [ 0.055472] x86: Booting SMP configuration: [ 0.056025] .... node #0, CPUs: #1 #2 #3 [ 0.060564] smp: Brought up 1 node, 4 CPUs [ 0.062017] smpboot: Max logical packages: 1 [ 0.063011] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.106646] node 0 deferred pages initialised in 41ms [ 0.109144] devtmpfs: initialized [ 0.110202] x86/mm: Memory block size: 128MB [ 0.112959] gcov: version magic: 0x41383552 [ 0.114527] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.115065] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.116198] pinctrl core: initialized pinctrl subsystem [ 0.117141] [ 0.117532] ************************************************************* [ 0.118010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.119011] ** ** [ 0.120011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.121009] ** ** [ 0.122011] ** This means that this kernel is built to expose internal ** [ 0.123010] ** IOMMU data structures, which may compromise security on ** [ 0.124010] ** your system. ** [ 0.125011] ** ** [ 0.126010] ** If you see this message and you are not debugging the ** [ 0.127009] ** kernel, report this immediately to your vendor! ** [ 0.128013] ** ** [ 0.129013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.130011] ************************************************************* [ 0.131695] NET: Registered protocol family 16 [ 0.132460] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.133060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.134102] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.135640] cpuidle: using governor menu [ 0.137947] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.140521] PCI: Using configuration type 1 for base access [ 0.143138] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.157584] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.158013] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.160068] cryptd: max_cpu_qlen set to 1000 [ 0.162699] ACPI: Added _OSI(Module Device) [ 0.163011] ACPI: Added _OSI(Processor Device) [ 0.164009] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.165009] ACPI: Added _OSI(Processor Aggregator Device) [ 0.168985] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.172336] ACPI: Interpreter enabled [ 0.173055] ACPI: PM: (supports S0 S3 S4 S5) [ 0.174014] ACPI: Using IOAPIC for interrupt routing [ 0.175125] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.176392] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.186505] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.187030] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.188020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.189073] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.191000] acpiphp: Slot [2] registered [ 0.192097] acpiphp: Slot [5] registered [ 0.194106] acpiphp: Slot [6] registered [ 0.196111] acpiphp: Slot [7] registered [ 0.197092] acpiphp: Slot [8] registered [ 0.199086] acpiphp: Slot [9] registered [ 0.201087] acpiphp: Slot [10] registered [ 0.202084] acpiphp: Slot [3] registered [ 0.204076] acpiphp: Slot [4] registered [ 0.205094] acpiphp: Slot [11] registered [ 0.207076] acpiphp: Slot [12] registered [ 0.208140] acpiphp: Slot [13] registered [ 0.209088] acpiphp: Slot [14] registered [ 0.211104] acpiphp: Slot [15] registered [ 0.213108] acpiphp: Slot [16] registered [ 0.215333] acpiphp: Slot [17] registered [ 0.218127] acpiphp: Slot [18] registered [ 0.220122] acpiphp: Slot [19] registered [ 0.223284] acpiphp: Slot [20] registered [ 0.227194] acpiphp: Slot [21] registered [ 0.229128] acpiphp: Slot [22] registered [ 0.230137] acpiphp: Slot [23] registered [ 0.232156] acpiphp: Slot [24] registered [ 0.234213] acpiphp: Slot [25] registered [ 0.236153] acpiphp: Slot [26] registered [ 0.238117] acpiphp: Slot [27] registered [ 0.239094] acpiphp: Slot [28] registered [ 0.240204] acpiphp: Slot [29] registered [ 0.242209] acpiphp: Slot [30] registered [ 0.244182] acpiphp: Slot [31] registered [ 0.246143] PCI host bridge to bus 0000:00 [ 0.248059] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.251034] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.253053] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.256027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.260069] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x38007fffffff window] [ 0.263065] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.266432] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.270533] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.275121] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.291027] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.297100] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.299038] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.302023] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.306027] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.308626] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.311815] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.314066] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.317821] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.323025] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.339021] pci 0000:00:02.0: reg 0x20: [mem 0x380000000000-0x380000003fff 64bit pref] [ 0.345022] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.360336] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.371025] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.378022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.401044] pci 0000:00:05.0: reg 0x20: [mem 0x380000004000-0x380000007fff 64bit pref] [ 0.415217] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.424092] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.429038] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.441021] pci 0000:00:06.0: reg 0x20: [mem 0x380000008000-0x38000000bfff 64bit pref] [ 0.465144] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.479029] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.488026] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.502030] pci 0000:00:07.0: reg 0x20: [mem 0x38000000c000-0x38000000ffff 64bit pref] [ 0.522907] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.529026] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.537017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.554045] pci 0000:00:08.0: reg 0x20: [mem 0x380000010000-0x380000013fff 64bit pref] [ 0.561000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.565015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.570033] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.592026] pci 0000:00:09.0: reg 0x20: [mem 0x380000014000-0x380000017fff 64bit pref] [ 0.613000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.621016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.633018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.659016] pci 0000:00:0a.0: reg 0x20: [mem 0x380000018000-0x38000001bfff 64bit pref] [ 0.671418] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.673397] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.675363] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.678360] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.680258] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.685090] iommu: Default domain type: Passthrough [ 0.686792] SCSI subsystem initialized [ 0.691180] ACPI: bus type USB registered [ 0.693111] usbcore: registered new interface driver usbfs [ 0.698156] usbcore: registered new interface driver hub [ 0.700090] usbcore: registered new device driver usb [ 0.703228] pps_core: LinuxPPS API ver. 1 registered [ 0.707016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.715033] PTP clock support registered [ 0.721026] EDAC MC: Ver: 3.0.0 [ 0.726189] PCI: Using ACPI for IRQ routing [ 0.728192] NetLabel: Initializing [ 0.730009] NetLabel: domain hash size = 128 [ 0.731009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.733132] NetLabel: unlabeled traffic allowed by default [ 0.735155] vgaarb: loaded [ 0.737312] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.739010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.745601] clocksource: Switched to clocksource kvm-clock [ 0.887917] VFS: Disk quotas dquot_6.6.0 [ 0.889581] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.892389] *** VALIDATE ramfs *** [ 0.893658] *** VALIDATE hugetlbfs *** [ 0.895267] pnp: PnP ACPI init [ 0.898623] pnp: PnP ACPI: found 6 devices [ 0.914038] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.917070] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.919236] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.921474] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.923802] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.926065] pci_bus 0000:00: resource 8 [mem 0x380000000000-0x38007fffffff window] [ 0.928789] NET: Registered protocol family 2 [ 0.931071] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.936061] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.939337] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.945238] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.948759] TCP: Hash tables configured (established 65536 bind 65536) [ 0.951683] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.954733] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.957408] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.960687] NET: Registered protocol family 1 [ 0.963885] RPC: Registered named UNIX socket transport module. [ 0.966310] RPC: Registered udp transport module. [ 0.967910] RPC: Registered tcp transport module. [ 0.969651] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.972883] NET: Registered protocol family 44 [ 0.974432] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.976638] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.978567] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.980765] PCI: CLS 0 bytes, default 64 [ 0.982864] Unpacking initramfs... [ 3.011924] debug: unmapping init [mem 0xffff900afcc54000-0xffff900afffbffff] [ 3.017337] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.019704] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.022602] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.659742] Initialise system trusted keyrings [ 3.661252] Key type blacklist registered [ 3.663731] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.672159] zbud: loaded [ 3.674990] *** VALIDATE nfs *** [ 3.676309] *** VALIDATE nfs4 *** [ 3.678137] pstore: using deflate compression [ 3.681676] Platform Keyring initialized [ 3.852392] NET: Registered protocol family 38 [ 3.856115] Key type asymmetric registered [ 3.859699] Asymmetric key parser 'x509' registered [ 3.864133] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.867706] io scheduler mq-deadline registered [ 3.869431] io scheduler kyber registered [ 3.871198] io scheduler bfq registered [ 3.873319] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.876493] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.879161] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.882254] ACPI: Power Button [PWRF] [ 3.983518] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.080297] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.295541] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.415626] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.620382] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.676119] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.732239] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.738880] Non-volatile memory driver v1.3 [ 4.743176] Linux agpgart interface v0.103 [ 4.790446] virtio_blk virtio1: [vda] 133928 512-byte logical blocks (68.6 MB/65.4 MiB) [ 4.793310] vda: detected capacity change from 0 to 68571136 [ 4.831431] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.837143] vdb: detected capacity change from 0 to 1073741824 [ 4.874967] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.880858] vdc: detected capacity change from 0 to 2621440000 [ 4.906609] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.910736] vdd: detected capacity change from 0 to 2621440000 [ 4.939230] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.943206] vde: detected capacity change from 0 to 4294967296 [ 4.967296] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.970815] vdf: detected capacity change from 0 to 4294967296 [ 4.980724] libphy: Fixed MDIO Bus: probed [ 4.990756] usbcore: registered new interface driver usbserial_generic [ 4.993221] usbserial: USB Serial support registered for generic [ 4.997811] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.005641] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.007340] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.011345] mousedev: PS/2 mouse device common for all mice [ 5.014883] rtc_cmos 00:05: RTC can wake from S4 [ 5.024415] rtc_cmos 00:05: registered as rtc0 [ 5.026479] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.030688] intel_pstate: CPU model not supported [ 5.036588] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.039848] hid: raw HID events driver (C) Jiri Kosina [ 5.044782] usbcore: registered new interface driver usbhid [ 5.049862] usbhid: USB HID core driver [ 5.052862] drop_monitor: Initializing network drop monitor service [ 5.055431] Initializing XFRM netlink socket [ 5.058515] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.060399] NET: Registered protocol family 10 [ 5.075770] Segment Routing with IPv6 [ 5.077147] NET: Registered protocol family 17 [ 5.079549] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.087568] mpls_gso: MPLS GSO support [ 5.094768] RAS: Correctable Errors collector initialized. [ 5.096472] AVX version of gcm_enc/dec engaged. [ 5.097629] AES CTR mode by8 optimization enabled [ 5.235042] sched_clock: Marking stable (5235022550, 0)->(6342468923, -1107446373) [ 5.245447] registered taskstats version 1 [ 5.247515] Loading compiled-in X.509 certificates [ 5.253558] zswap: loaded using pool lzo/zbud [ 5.297988] Key type big_key registered [ 5.317128] Key type encrypted registered [ 5.325839] ima: No TPM chip found, activating TPM-bypass! [ 5.329705] ima: Allocated hash algorithm: sha1 [ 5.330978] ima: No architecture policies found [ 5.332470] evm: Initialising EVM extended attributes: [ 5.335117] evm: security.selinux [ 5.339543] evm: security.ima [ 5.340734] evm: security.capability [ 5.345405] evm: HMAC attrs: 0x1 [ 5.347866] rtc_cmos 00:05: setting system clock to 2025-12-03 03:39:58 UTC (1764733198) [ 5.358199] debug: unmapping init [mem 0xffffffff92a03000-0xffffffff92bfffff] [ 5.363913] debug: unmapping init [mem 0xffffffff91782000-0xffffffff91a58fff] [ 5.376630] Write protecting the kernel read-only data: 28672k [ 5.380378] debug: unmapping init [mem 0xffffffff8fe03000-0xffffffff8fffffff] [ 5.386426] debug: unmapping init [mem 0xffffffff90714000-0xffffffff907fffff] [ 5.456916] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.466609] systemd[1]: Detected virtualization kvm. [ 5.471537] systemd[1]: Detected architecture x86-64. [ 5.473389] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.500274] systemd[1]: No hostname configured. [ 5.502184] systemd[1]: Set hostname to . [ 5.504189] random: systemd: uninitialized urandom read (16 bytes read) [ 5.509898] systemd[1]: Initializing machine ID from random generator. [ 5.642339] random: ln: uninitialized urandom read (6 bytes read) [ 5.865115] random: systemd: uninitialized urandom read (16 bytes read) [ 5.867616] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 5.880868] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 5.885906] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.226471] device-mapper: uevent: version 1.0.3 [ 7.228433] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 8.592402] random: fast init done [ 8.636932] virtio_net virtio0 ens2: renamed from eth0 [ 9.024828] scsi host0: ata_piix [ 9.108129] scsi host1: ata_piix [ 9.109686] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 9.114812] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 14.047203] random: crng init done [ 14.048788] random: 7 urandom warning(s) missed due to ratelimiting [ 16.643900] dracut-initqueue[585]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 18.718769] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 20.585701] printk: systemd: 26 output lines suppressed due to ratelimiting [ 21.086411] SELinux: Disabled at runtime. [ 21.160048] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 21.168788] systemd[1]: Detected virtualization kvm. [ 21.171387] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 22.239668] systemd[1]: initrd-switch-root.service: Succeeded. [ 22.249713] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 22.270589] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 22.283612] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 22.289739] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 22.308106] systemd[1]: Starting Journal Service... Starting Journal Service... [ 22.326589] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. Mounting POSIX Message Queue File System... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Kernel Debug File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target rpc_pipefs.target. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. Mounting Huge Pages File System... [ OK ] Created slice system-getty.slice. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-serial\x2dgetty.slice. [ 22.585390] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 23.234284] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 24.012158] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 24.071167] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 24.834303] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 24.882909] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (6s / no limit) [** ] A start job is running for Configur…-only root support (6s / no limit)[ 29.323839] Key type dns_resolver registered [*** ] A start job is running for Configur…-only root support (7s / no limit) [ *** ] A start job is running for Configur…-only root support (8s / no limit)[ 30.311770] NFS: Registering the id_resolver key type [ 30.313862] Key type id_resolver registered [ 30.315472] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started Login Service. [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg148-server login: [ 67.563012] hrtimer: interrupt took 4392774 ns [ 90.115733] libcfs: loading out-of-tree module taints kernel. [ 90.144948] Key type ._llcrypt registered [ 90.146263] Key type .llcrypt registered [ 90.275799] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 109.404259] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 110.621587] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 110.636226] alg: No test for adler32 (adler32-zlib) [ 112.077907] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 112.646383] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 114.383184] Key type lgssc registered [ 115.750907] Lustre: Echo OBD driver; http://www.lustre.org/ [ 129.812690] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 131.280526] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 138.280359] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 144.478861] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 150.226502] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 162.532088] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 172.499253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 172.557970] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 172.572607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 173.817741] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 173.849904] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 173.959387] Lustre: lustre-MDT0000: new disk, initializing [ 174.034145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 174.064270] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 177.009489] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 187.554354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 187.627497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 187.683265] Lustre: 6514:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 187.721392] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 187.727577] Lustre: Skipped 1 previous similar message [ 187.815747] Lustre: lustre-MDT0001: new disk, initializing [ 187.886363] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 187.904055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 187.918773] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 190.861351] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 194.939585] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 202.935868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 203.002237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 203.205251] Lustre: lustre-OST0000: new disk, initializing [ 203.207893] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 203.250766] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 203.654466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 203.659232] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 203.736704] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 207.735806] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 222.165389] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 222.212570] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 222.307748] Lustre: lustre-OST0001: new disk, initializing [ 222.316621] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 222.421163] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 231.499535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 231.525119] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 231.662491] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 231.730957] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 247.991377] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 255.988588] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 267.507958] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing check_logdir /tmp/testlogs/ [ 272.287324] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing yml_node [ 276.985466] Lustre: DEBUG MARKER: Client: 2.17.0.RC1 [ 279.580516] Lustre: DEBUG MARKER: MDS: 2.17.0.RC1 [ 281.943738] Lustre: DEBUG MARKER: OSS: 2.17.0.RC1 [ 283.826757] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Dec 2 22:44:35 EST 2025 [ 302.291840] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 303.963146] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 318.951702] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 318.957272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 318.968232] Lustre: Skipped 2 previous similar messages [ 318.971041] Lustre: Skipped 1 previous similar message [ 324.065385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 324.065385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 324.065398] Lustre: Skipped 2 previous similar messages [ 324.071784] Lustre: Skipped 4 previous similar messages [ 324.802481] LustreError: 12219:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 324.863625] Lustre: server umount lustre-MDT0000 complete [ 327.659589] LustreError: 10340:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764733520 with bad export cookie 15687622452202556650 [ 327.666679] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 327.706848] LustreError: 10340:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 329.184543] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 329.187639] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 329.202059] Lustre: Skipped 2 previous similar messages [ 333.914952] LustreError: 12422:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 333.919025] LustreError: 12422:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 334.138736] Lustre: server umount lustre-MDT0001 complete [ 337.805433] LustreError: 12625:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 337.839989] LustreError: 12625:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 337.977293] Lustre: server umount lustre-OST0000 complete [ 341.355672] LustreError: 12826:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 341.369521] LustreError: 12826:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 341.515760] Lustre: server umount lustre-OST0001 complete [ 347.151503] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 355.870890] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 365.431825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 371.668843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 378.144211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 384.638936] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 391.476151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 391.532778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 391.734734] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 391.776912] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 391.832753] Lustre: lustre-MDT0000: new disk, initializing [ 391.929908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 391.943488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 395.779550] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 406.913970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 406.996990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 407.104965] Lustre: 15008:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 407.201288] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 407.212386] Lustre: Skipped 1 previous similar message [ 407.365903] Lustre: lustre-MDT0001: new disk, initializing [ 407.510227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 407.566221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 407.585796] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 411.704597] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 419.183346] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 423.115513] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 429.636709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 429.731718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 429.992758] Lustre: lustre-OST0000: new disk, initializing [ 429.999983] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 430.089959] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 431.496814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 431.511467] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 431.630229] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 435.827575] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 442.980445] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 448.357333] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 448.740908] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 452.918676] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 453.131778] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 457.190691] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 457.191494] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 457.201055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 457.201068] Lustre: Skipped 3 previous similar messages [ 457.205301] Lustre: Skipped 1 previous similar message [ 457.220229] LustreError: Skipped 1 previous similar message [ 461.442335] LustreError: 17950:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 461.458268] LustreError: 17950:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 461.608217] Lustre: server umount lustre-OST0000 complete [ 467.430702] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 467.433250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 467.461939] Lustre: Skipped 1 previous similar message [ 467.490158] Lustre: Skipped 2 previous similar messages [ 471.915993] LustreError: 18152:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 471.931803] LustreError: 18152:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 472.045539] Lustre: server umount lustre-MDT0000 complete [ 472.559200] LustreError: 15022:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 472.587314] LustreError: 15022:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 475.842218] LustreError: 15003:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764733668 with bad export cookie 15687622452202558183 [ 475.844490] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 475.864095] LustreError: 15003:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 476.299355] Lustre: server umount lustre-MDT0001 complete [ 482.220494] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 22:47:54 (1764733674) [ 498.130844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 498.686197] LustreError: 19489:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 498.784309] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 502.440226] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 503.778510] LustreError: 19489:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 507.883486] LustreError: 19489:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 510.708252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 515.213673] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 521.105876] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 524.532531] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 526.602220] Lustre: Failing over lustre-MDT0000 [ 526.759270] LustreError: 20832:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 526.764852] LustreError: 20832:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 526.854209] Lustre: server umount lustre-MDT0000 complete [ 528.868371] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 528.870721] LustreError: 20201:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 528.889701] Lustre: Skipped 1 previous similar message [ 528.907759] LustreError: 20201:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 541.152489] LustreError: 20201:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 541.170471] LustreError: 20201:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 545.055101] Lustre: 3659:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764733722/real 1764733722] req@ffff900b80c81880 x1850456990182144/t0(0) o400->MGC192.168.201.148@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1764733738 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 545.099213] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 545.372849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 555.488576] LustreError: 3655:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff900b5225c380 x1850456990184704/t0(0) o250->MGC192.168.201.148@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 556.709835] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 556.712336] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 556.737019] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 558.922496] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 565.912389] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 571.360980] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 571.366080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 571.375197] Lustre: Skipped 1 previous similar message [ 575.700934] LustreError: 21812:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 575.710066] LustreError: 21812:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 575.811674] Lustre: server umount lustre-MDT0000 complete [ 576.482641] LustreError: 19488:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 576.490237] LustreError: 19488:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 578.531963] LustreError: 19468:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764733771 with bad export cookie 15687622452202559597 [ 578.539883] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 578.540907] LustreError: 19468:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 578.750029] Lustre: server umount lustre-MDT0001 complete [ 585.242716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 585.768623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 585.782205] Lustre: Skipped 2 previous similar messages [ 588.514566] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 594.467529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 597.588571] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 601.955597] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 605.289336] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 607.259567] Lustre: Failing over lustre-MDT0000 [ 607.417415] Lustre: server umount lustre-MDT0000 complete [ 609.252876] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 609.266211] LustreError: 22487:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 609.309057] LustreError: 22487:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 625.435195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 625.562589] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 625.864458] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 628.729112] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 631.267271] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 631.272218] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 631.283542] Lustre: Skipped 1 previous similar message [ 631.307831] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 634.645743] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 638.883368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 638.892303] Lustre: Skipped 2 previous similar messages [ 652.255493] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 652.346244] LustreError: 24812:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 652.348814] LustreError: 24812:0:(obd_class.h:479:obd_check_dev()) Skipped 19 previous similar messages [ 652.443507] Lustre: server umount lustre-MDT0000 complete [ 655.869500] LustreError: 23834:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764733849 with bad export cookie 15687622452202560528 [ 655.875467] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 655.890374] LustreError: 23834:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 656.189169] Lustre: server umount lustre-MDT0001 complete [ 664.029723] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 22:50:56 (1764733856) [ 676.190651] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 685.767453] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 695.027764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 701.333123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 707.195474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 712.856301] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 720.139428] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 728.344886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 728.380768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 728.522957] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 728.562317] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 728.613809] Lustre: lustre-MDT0000: new disk, initializing [ 728.707097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 728.710490] Lustre: Skipped 2 previous similar messages [ 728.721617] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 732.671525] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 743.734284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 743.782947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 743.845609] Lustre: 28484:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 744.079482] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 747.238151] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 754.354863] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 757.956780] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 764.094428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 764.160199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 765.682307] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 765.739738] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 765.841352] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 765.851925] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 769.294296] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 774.307771] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 777.873171] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 796.646038] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 796.646988] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 796.656327] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 796.656814] Lustre: Skipped 4 previous similar messages [ 796.659214] Lustre: Skipped 4 previous similar messages [ 798.742190] LustreError: 31158:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 798.745647] LustreError: 31158:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 798.833342] Lustre: server umount lustre-OST0000 complete [ 807.436364] Lustre: server umount lustre-MDT0000 complete [ 810.378268] LustreError: 28475:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764734003 with bad export cookie 15687622452202560927 [ 810.388037] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 810.705168] Lustre: server umount lustre-MDT0001 complete [ 817.598860] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 819.733152] Key type lgssc unregistered [ 819.942573] LNet: 32091:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 819.949305] LNetError: 32091:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 819.971698] LNet: Removed LNI 192.168.201.148@tcp [ 820.676376] Key type .llcrypt unregistered [ 820.678240] Key type ._llcrypt unregistered [ 837.484632] Key type ._llcrypt registered [ 837.486250] Key type .llcrypt registered [ 837.554658] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 838.185775] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 838.198306] alg: No test for adler32 (adler32-zlib) [ 839.155353] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 839.305727] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 840.935221] Key type lgssc registered [ 841.662861] Lustre: Echo OBD driver; http://www.lustre.org/ [ 846.976088] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 846.994403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 848.309454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 850.869991] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 856.875294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 857.094977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 859.551084] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 863.256829] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 865.358577] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 869.701132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 869.958482] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 873.086198] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 873.958099] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 877.817655] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 882.285166] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 892.187746] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 22:54:44 (1764734084) [ 893.398621] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 894.677499] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 898.977242] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 22:54:51 (1764734091) [ 904.814436] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 22:54:57 (1764734097) [ 905.533684] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 911.711144] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 911.714675] Lustre: Skipped 2 previous similar messages [ 912.944178] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 22:55:05 (1764734105) [ 915.797010] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 915.799295] Lustre: Skipped 3 previous similar messages [ 926.803313] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 926.807110] Lustre: Skipped 6 previous similar messages [ 928.459512] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 22:55:21 (1764734121) [ 930.548802] Lustre: *** cfs_fail_loc=131b, val=2*** [ 933.346382] Lustre: *** cfs_fail_loc=131b, val=2*** [ 940.332308] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 22:55:32 (1764734132) [ 943.509639] Lustre: Setting parameter general.jobid_name=TESTNAME1 in log params [ 943.513544] Lustre: Skipped 5 previous similar messages [ 950.461332] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 22:55:43 (1764734143) [ 964.576126] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 964.579574] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 964.592980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 964.665777] Lustre: Skipped 1 previous similar message [ 964.792828] Lustre: Skipped 1 previous similar message [ 968.725314] LustreError: 40017:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 968.979907] Lustre: server umount lustre-OST0000 complete [ 974.394724] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 974.421523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 974.430912] Lustre: Skipped 1 previous similar message [ 979.429410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 979.451826] Lustre: Skipped 1 previous similar message [ 980.659624] LustreError: 40220:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 980.676888] LustreError: 40220:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 980.804293] Lustre: server umount lustre-MDT0000 complete [ 984.545062] LustreError: 33463:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 984.566025] LustreError: 33463:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 985.189140] LustreError: 33442:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764734178 with bad export cookie 1044401625412487083 [ 985.215213] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 985.220690] LustreError: 33442:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 985.404920] LustreError: 40421:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 985.410067] LustreError: 40421:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 985.715340] Lustre: server umount lustre-MDT0001 complete [ 997.698288] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 1000.488184] Key type lgssc unregistered [ 1000.815691] LNet: 40953:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1000.829217] LNetError: 40953:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1000.858479] LNet: Removed LNI 192.168.201.148@tcp [ 1001.717756] Key type .llcrypt unregistered [ 1001.719556] Key type ._llcrypt unregistered [ 1017.356530] Key type ._llcrypt registered [ 1017.364363] Key type .llcrypt registered [ 1017.479881] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 1031.019190] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 1032.106342] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1032.162768] alg: No test for adler32 (adler32-zlib) [ 1033.308912] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 1033.639894] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 1035.311205] Key type lgssc registered [ 1036.304969] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1043.339896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1047.684854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1052.525882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1057.652386] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1062.725801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1062.770623] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1062.783991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1064.011767] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1064.057390] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1064.147784] Lustre: lustre-MDT0000: new disk, initializing [ 1064.231392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1064.251752] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1067.467644] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1076.695453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1076.737960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1076.797929] Lustre: 44081:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1076.836476] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1076.839835] Lustre: Skipped 1 previous similar message [ 1076.896092] Lustre: lustre-MDT0001: new disk, initializing [ 1076.974874] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1077.010050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1077.017389] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1080.373287] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1086.873100] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1090.225135] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1095.897445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1095.959626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1096.190389] Lustre: lustre-OST0000: new disk, initializing [ 1096.195663] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1096.284289] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1101.110943] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1103.908744] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1103.919051] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1104.021082] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1107.344453] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1111.151389] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1111.311994] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1115.002179] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1115.158299] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1119.714374] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1119.718408] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1119.738629] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1122.951285] LustreError: 47019:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1123.056699] Lustre: server umount lustre-OST0000 complete [ 1128.417429] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1128.427592] LustreError: Skipped 1 previous similar message [ 1128.434351] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1128.449521] Lustre: Skipped 1 previous similar message [ 1128.452844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1128.458459] Lustre: Skipped 1 previous similar message [ 1129.955191] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1129.955254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1129.979021] Lustre: Skipped 1 previous similar message [ 1132.838671] LustreError: 47220:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1132.841885] LustreError: 47220:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1132.911825] Lustre: server umount lustre-MDT0000 complete [ 1135.791801] LustreError: 44072:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764734328 with bad export cookie 11233664390156110480 [ 1135.793635] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1135.804596] LustreError: 44072:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1135.929316] LustreError: 47421:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1135.933223] LustreError: 47421:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1136.061366] Lustre: server umount lustre-MDT0001 complete [ 1143.396411] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 22:58:55 (1764734335) [ 1148.776242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1149.109205] LustreError: 48141:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1149.122661] LustreError: 48141:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1149.171638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1151.970311] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1154.530543] LustreError: 48141:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1158.480879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1158.766771] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1162.658769] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1167.826385] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1171.247018] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1176.462745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1176.769574] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1181.464583] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1185.651128] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1199.752966] LustreError: 50360:0:(llog.c:441:llog_init_handle()) llog type is not specified! [ 1200.604154] Lustre: Modifying parameter general.timeout=1 in log params [ 1202.207853] Lustre: Modifying parameter general.timeout=3 in log params [ 1202.211731] Lustre: Skipped 1 previous similar message [ 1204.387208] Lustre: Modifying parameter general.timeout=6 in log params [ 1204.389921] Lustre: Skipped 2 previous similar messages [ 1208.925288] Lustre: Modifying parameter general.timeout=12 in log params [ 1208.928126] Lustre: Skipped 5 previous similar messages [ 1217.502723] Lustre: Modifying parameter general.timeout=25 in log params [ 1217.512168] Lustre: Skipped 12 previous similar messages [ 1233.857130] Lustre: Modifying parameter general.timeout=47 in log params [ 1233.859838] Lustre: Skipped 21 previous similar messages [ 1266.095937] Lustre: Modifying parameter general.timeout=91 in log params [ 1266.097902] Lustre: Skipped 43 previous similar messages [ 1304.921798] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 23:01:36 (1764734496) [ 1308.131572] Lustre: 57304:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1310.170454] Lustre: 57403:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1313.644373] Lustre: 57612:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1313.651970] Lustre: 57612:0:(mgs_llog.c:1346:mgs_modify_param()) Skipped 1 previous similar message [ 1330.582039] Lustre: Disabling parameter general.ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=50 in log params [ 1330.587579] Lustre: Skipped 55 previous similar messages [ 1340.054279] Lustre: 59179:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1340.059381] Lustre: 59179:0:(mgs_llog.c:1346:mgs_modify_param()) Skipped 2 previous similar messages [ 1340.072777] LustreError: 59179:0:(mgs_handler.c:1126:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1343.246655] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 23:02:15 (1764734535) [ 1352.161451] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1352.165561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1352.173248] Lustre: Skipped 2 previous similar messages [ 1352.176349] Lustre: Skipped 1 previous similar message [ 1357.282456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1357.288136] Lustre: Skipped 3 previous similar messages [ 1357.481753] LustreError: 59878:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1357.485782] LustreError: 59878:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1357.587101] Lustre: server umount lustre-MDT0000 complete [ 1360.492169] LustreError: 48121:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764734553 with bad export cookie 11233664390156111362 [ 1360.494671] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1360.567125] LustreError: 48121:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1360.804117] Lustre: server umount lustre-MDT0001 complete [ 1372.936969] LustreError: 60280:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1372.939698] LustreError: 60280:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1373.124648] Lustre: server umount lustre-OST0000 complete [ 1380.555036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1383.741887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1386.856274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1389.687206] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1394.363735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1394.573366] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1394.783171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1397.490588] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1404.164230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1404.270594] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1404.314229] Lustre: 61732:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1407.421974] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1411.526393] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1413.988158] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1418.709269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1418.835403] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1423.080381] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1427.022314] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1433.713226] Lustre: 63963:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1439.208983] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1439.215220] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1439.230293] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1439.238960] Lustre: Skipped 1 previous similar message [ 1444.151807] LustreError: 64208:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1444.157556] LustreError: 64208:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1444.244768] Lustre: server umount lustre-OST0000 complete [ 1449.444052] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1449.450263] Lustre: Skipped 2 previous similar messages [ 1453.196920] Lustre: server umount lustre-MDT0000 complete [ 1454.563590] LustreError: 61743:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1454.573461] LustreError: 61743:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1455.618116] LustreError: 63368:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764734648 with bad export cookie 11233664390156114183 [ 1455.624771] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1455.856452] Lustre: server umount lustre-MDT0001 complete [ 1462.926490] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 1465.311696] Key type lgssc unregistered [ 1465.586732] LNet: 65152:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1465.596598] LNetError: 65152:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1466.661615] LNet: Removed LNI 192.168.201.148@tcp [ 1467.327217] Key type .llcrypt unregistered [ 1467.329035] Key type ._llcrypt unregistered [ 1485.590231] Key type ._llcrypt registered [ 1485.592528] Key type .llcrypt registered [ 1485.677880] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 1486.192592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1486.211858] alg: No test for adler32 (adler32-zlib) [ 1487.204407] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 1487.359652] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 1489.031229] Key type lgssc registered [ 1489.912965] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1495.836651] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1495.856638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1497.320810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1500.070658] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1506.879885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1507.360022] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1510.430544] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1514.325080] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1517.576285] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1522.476905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1522.894285] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1526.619774] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1530.492898] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1537.329984] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1547.746423] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1547.748080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1547.752112] Lustre: Skipped 1 previous similar message [ 1547.757395] Lustre: Skipped 2 previous similar messages [ 1549.361745] LustreError: 69231:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1549.412775] Lustre: server umount lustre-MDT0000 complete [ 1552.390968] LustreError: 66506:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764734745 with bad export cookie 15835030094993626351 [ 1552.397260] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1552.398974] LustreError: 66506:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1552.544503] LustreError: 69434:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1552.553605] LustreError: 69434:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1552.705461] Lustre: server umount lustre-MDT0001 complete [ 1566.552917] LustreError: 69635:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1566.558810] LustreError: 69635:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1566.663419] Lustre: server umount lustre-OST0000 complete [ 1573.222896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1576.464139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1579.713484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1582.807497] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1587.832846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1588.015681] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1588.061831] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1588.276542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1591.164382] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1598.097905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1598.190984] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1598.216201] Lustre: 71087:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1598.614819] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1601.574143] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1605.466428] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1608.211511] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1613.109254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1613.245597] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1613.519598] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1617.366411] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1621.115112] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1627.585490] Lustre: 73327:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1628.750083] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1633.254089] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1633.254398] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1633.255855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1633.259128] Lustre: Skipped 2 previous similar messages [ 1633.264617] LustreError: Skipped 1 previous similar message [ 1637.559169] LustreError: 73587:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1637.562091] LustreError: 73587:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1637.631066] Lustre: server umount lustre-OST0000 complete [ 1643.493451] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1643.505361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1643.508340] Lustre: Skipped 2 previous similar messages [ 1648.614310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1648.617328] Lustre: Skipped 1 previous similar message [ 1653.729273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1653.732381] Lustre: Skipped 1 previous similar message [ 1654.239166] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1654.315724] LustreError: 73789:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1654.318994] LustreError: 73789:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1654.357442] Lustre: server umount lustre-MDT0000 complete [ 1656.656721] LustreError: 73331:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764734849 with bad export cookie 15835030094993627184 [ 1656.660599] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1656.663611] LustreError: 73331:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1656.864103] Lustre: server umount lustre-MDT0001 complete [ 1662.703173] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 1664.477754] Key type lgssc unregistered [ 1664.676449] LNet: 74523:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1664.681301] LNetError: 74523:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1664.690174] LNet: Removed LNI 192.168.201.148@tcp [ 1665.216486] Key type .llcrypt unregistered [ 1665.218090] Key type ._llcrypt unregistered [ 1679.635601] Key type ._llcrypt registered [ 1679.637440] Key type .llcrypt registered [ 1679.692188] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 1680.133031] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1680.195162] alg: No test for adler32 (adler32-zlib) [ 1681.079251] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 1681.195104] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 1682.799314] Key type lgssc registered [ 1683.490453] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1687.847841] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1687.862894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1689.154098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1691.282946] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1695.979506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1696.321624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1698.436039] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1701.625731] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1703.962698] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1708.199746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1708.553638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1711.985915] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1715.244512] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1724.414785] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1725.020342] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1726.117908] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1726.121714] Lustre: Skipped 1 previous similar message [ 1728.377858] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1728.381121] Lustre: Skipped 3 previous similar messages [ 1732.786149] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1732.789221] Lustre: Skipped 7 previous similar messages [ 1741.244495] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1741.248112] Lustre: Skipped 15 previous similar messages [ 1757.667369] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1757.670626] Lustre: Skipped 30 previous similar messages [ 1784.041410] Lustre: 83646:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1785.931657] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 23:09:38 (1764734978) [ 1786.960457] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1788.594830] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 23:09:40 (1764734980) [ 1791.457539] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1791.461792] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1791.470811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1792.480527] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1792.484308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1792.490746] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1797.601807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1797.609491] Lustre: Skipped 1 previous similar message [ 1802.722805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1802.729116] Lustre: Skipped 1 previous similar message [ 1805.280085] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1805.317144] LustreError: 83942:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1805.389024] Lustre: server umount lustre-OST0000 complete [ 1807.841567] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1807.843569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1807.849417] Lustre: Skipped 1 previous similar message [ 1818.081860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1818.088181] Lustre: Skipped 4 previous similar messages [ 1822.175133] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1822.269988] LustreError: 84144:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1822.275638] LustreError: 84144:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1822.345150] Lustre: server umount lustre-MDT0000 complete [ 1823.201175] LustreError: 75891:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1823.214573] LustreError: 75891:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1824.629035] LustreError: 75876:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764735017 with bad export cookie 13890834410574799895 [ 1824.632169] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1824.636278] LustreError: 75876:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1824.725782] LustreError: 84346:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1824.728653] LustreError: 84346:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1824.834951] Lustre: server umount lustre-MDT0001 complete [ 1831.356142] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 1833.235532] Key type lgssc unregistered [ 1833.427868] LNet: 84878:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1833.434983] LNetError: 84878:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1833.445872] LNet: Removed LNI 192.168.201.148@tcp [ 1833.941187] Key type .llcrypt unregistered [ 1833.942682] Key type ._llcrypt unregistered [ 1836.364748] Key type ._llcrypt registered [ 1836.366078] Key type .llcrypt registered [ 1836.452950] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1839.226353] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules [ 1839.762481] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1839.833043] alg: No test for adler32 (adler32-zlib) [ 1840.770409] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 1840.917347] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 1842.575151] Key type lgssc registered [ 1843.285571] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1844.099074] LustreError: 86018:0:(lu_object.c:1438:lu_context_key_register()) cfs_race id 60d sleeping [ 1854.132693] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 1858.945290] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1864.664781] LustreError: 86018:0:(lu_object.c:1438:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1864.704299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1866.072113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1868.553113] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1874.042860] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 23:11:06 (1764735066) [ 1876.457958] LustreError: 86999:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1876.469238] LustreError: 86999:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1876.934428] LustreError: 87031:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1876.938808] LustreError: 87031:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1876.943258] LustreError: 87031:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 11, retries 0, failed: rc = -5 [ 1877.057062] LustreError: 87805:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1877.173203] Lustre: server umount lustre-MDT0000 complete [ 1884.414474] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 1885.996374] Key type lgssc unregistered [ 1886.189663] LNet: 88490:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1886.194933] LNetError: 88490:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1886.205711] LNet: Removed LNI 192.168.201.148@tcp [ 1886.629145] Key type .llcrypt unregistered [ 1886.632200] Key type ._llcrypt unregistered [ 1896.384200] Key type ._llcrypt registered [ 1896.385655] Key type .llcrypt registered [ 1896.443386] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 1896.845694] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1896.887732] alg: No test for adler32 (adler32-zlib) [ 1897.822403] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 1897.946615] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 1899.584813] Key type lgssc registered [ 1900.211654] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1904.231488] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1904.242254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1905.524112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1907.430713] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1911.526263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1911.876528] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1913.709588] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1916.577304] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1918.492644] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1922.118847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1922.462185] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1925.265786] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 1927.951929] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1965.848951] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 23:12:38 (1764735158) [ 1966.744750] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1967.756410] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 23:12:40 (1764735160) [ 1972.703853] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1972.708577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1972.712263] Lustre: Skipped 2 previous similar messages [ 1972.715198] Lustre: Skipped 1 previous similar message [ 1975.434530] LustreError: 92595:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1975.477289] Lustre: server umount lustre-MDT0000 complete [ 1977.372135] LustreError: 89649:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764735170 with bad export cookie 18114272819006800017 [ 1977.374380] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1977.381526] LustreError: 89649:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1977.487031] LustreError: 92796:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1977.489810] LustreError: 92796:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1977.578546] Lustre: server umount lustre-MDT0001 complete [ 1989.704044] LustreError: 92996:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1989.708784] LustreError: 92996:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1989.880332] Lustre: server umount lustre-OST0000 complete [ 1996.404287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1996.654501] LustreError: 93626:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1996.662134] LustreError: 93626:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1996.791182] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1998.793832] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2001.888178] LustreError: 93626:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2003.073207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2003.413857] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2005.314661] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2008.080695] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2010.106539] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2014.014110] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2017.672559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2017.717800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2017.823922] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2017.828497] LustreError: 93614:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2017.842931] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2017.848439] LustreError: 95488:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -98 [ 2017.853691] LustreError: 95488:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 2017.857316] LustreError: 95488:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 2017.897276] Lustre: server umount lustre-OST0000 complete [ 2017.899296] LustreError: 95488:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2022.935446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2023.057641] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2023.061892] LustreError: 93615:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2023.066863] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2023.070957] LustreError: 95949:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -98 [ 2023.074161] LustreError: 95949:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 2023.076660] LustreError: 95949:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 2023.113572] Lustre: server umount lustre-OST0000 complete [ 2023.115635] LustreError: 95949:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2024.646837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2028.342797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2028.452986] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2028.456471] Lustre: Found index 0 for lustre-OST0000, updating log [ 2028.462320] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2028.502391] Lustre: lustre-OST0000: new disk, initializing [ 2028.505034] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2028.632571] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2031.138726] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2032.875030] Lustre: Failing over lustre-OST0000 [ 2032.898983] LustreError: 97034:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2032.901666] LustreError: 97034:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2032.952860] Lustre: server umount lustre-OST0000 complete [ 2038.752818] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2038.753300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2038.758160] Lustre: Skipped 1 previous similar message [ 2038.761142] Lustre: Skipped 1 previous similar message [ 2040.773519] LustreError: 97237:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2040.776831] LustreError: 97237:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2040.821286] Lustre: server umount lustre-MDT0000 complete [ 2042.685599] LustreError: 94352:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764735235 with bad export cookie 18114272819006802761 [ 2042.687073] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2042.692806] LustreError: 94352:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2042.867466] Lustre: server umount lustre-MDT0001 complete [ 2046.943882] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 23:13:59 (1764735239) [ 2054.414310] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 2059.457276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2059.699161] LustreError: 98805:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2059.770167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2061.772886] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2064.866598] LustreError: 98805:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2065.695860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2067.788347] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2069.236235] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2072.628738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2075.795574] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2080.014103] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2080.049720] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2080.086777] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2080.129185] Lustre: lustre-OST0001: new disk, initializing [ 2080.132232] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2082.671666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2082.676877] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2082.730790] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2083.017389] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2087.373617] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2091.810891] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2099.997448] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.48@tcp (stopping) [ 2100.000934] Lustre: Skipped 1 previous similar message [ 2100.193551] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2100.200871] Lustre: Skipped 2 previous similar messages [ 2100.203282] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2104.721181] LustreError: 102045:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 2104.723937] LustreError: 102045:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 2104.761613] Lustre: server umount lustre-MDT0001 complete [ 2105.107526] LustreError: 99411:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2106.079617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2109.174888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2109.239629] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2109.242087] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2109.265246] Lustre: 98794:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2109.597384] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2109.600223] Lustre: Skipped 3 previous similar messages [ 2111.272352] LustreError: 102797:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2111.276226] LustreError: 102797:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2111.281269] LustreError: 98797:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2111.328471] LustreError: 102817:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.201.148@tcp: cfg command failed: rc = -17 [ 2111.332842] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2111.332842] [ 2111.359606] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2115.050145] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2115.372727] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 23:15:08 (1764735308) [ 2122.725589] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 2131.111383] Lustre: 104653:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2150.650270] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2155.044152] Lustre: 105787:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2237.408689] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2237.419545] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2237.447708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2237.457933] Lustre: Skipped 3 previous similar messages [ 2240.740492] LustreError: 106245:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 2240.742833] LustreError: 106245:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2240.801113] Lustre: server umount lustre-MDT0000 complete [ 2243.041948] LustreError: 98799:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2243.067311] LustreError: 98799:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 2243.724360] LustreError: 101841:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764735436 with bad export cookie 18114272819006803671 [ 2243.728766] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2243.733359] LustreError: 101841:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2243.990159] Lustre: server umount lustre-MDT0001 complete [ 2257.494022] Lustre: server umount lustre-OST0000 complete [ 2270.734154] Lustre: server umount lustre-OST0001 complete [ 2274.855247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2280.618092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2281.456833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2287.895593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2294.410458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2295.352472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2308.893299] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 2316.156106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2316.172087] Lustre: lustre-MDT0000: reset Object Index mappings [ 2316.520298] LustreError: 109726:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2316.586063] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2319.451897] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2325.775273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2325.785271] Lustre: lustre-MDT0001: reset Object Index mappings [ 2329.084551] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2331.121623] Lustre: 110840:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2335.365263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2336.887110] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 2339.813223] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2340.984287] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 2345.538859] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2347.534891] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 2347.553651] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 2350.219716] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2355.968698] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2358.196276] Lustre: 112696:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2398.551637] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 23:19:51 (1764735591) [ 2427.877660] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2427.879626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2427.884470] Lustre: Skipped 6 previous similar messages [ 2427.891018] Lustre: Skipped 6 previous similar messages [ 2431.512592] LustreError: 113046:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2431.515624] LustreError: 113046:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 2431.559090] Lustre: server umount lustre-MDT0000 complete [ 2432.994470] LustreError: 111220:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2433.001391] LustreError: 111220:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2433.213464] LustreError: 109706:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764735626 with bad export cookie 18114272819006850522 [ 2433.219053] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2433.219248] LustreError: 109706:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2438.116489] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2438.118322] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2438.123498] Lustre: Skipped 1 previous similar message [ 2438.128553] Lustre: Skipped 1 previous similar message [ 2439.590498] Lustre: server umount lustre-MDT0001 complete [ 2447.973705] Lustre: server umount lustre-OST0000 complete [ 2464.223284] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2464.303800] Lustre: server umount lustre-OST0001 complete [ 2467.059394] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 2471.957320] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 2476.608566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2479.691927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2482.584151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2485.570985] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2488.758875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2488.787992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2488.907046] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2488.929784] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2488.973982] Lustre: lustre-MDT0000: new disk, initializing [ 2489.026315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2489.029501] Lustre: Skipped 3 previous similar messages [ 2489.042336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2490.826458] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2494.506873] Lustre: server umount lustre-MDT0000 complete [ 2495.819237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2498.996963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2499.050465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2499.186910] Lustre: Found index 0 for lustre-MDT0000, updating log [ 2499.197484] Lustre: 116918:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2499.207351] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 2500.988805] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2503.314659] LustreError: 117432:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2503.318351] LustreError: 117432:0:(obd_class.h:479:obd_check_dev()) Skipped 29 previous similar messages [ 2503.405269] Lustre: server umount lustre-MDT0000 complete [ 2507.536197] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 23:21:40 (1764735700) [ 2508.289412] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 2509.091396] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 23:21:42 (1764735702) [ 2516.600236] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 2521.228818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2523.145982] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2527.071260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2527.106110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2527.161437] Lustre: 118934:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2527.194712] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2527.197559] Lustre: Skipped 1 previous similar message [ 2527.238568] Lustre: lustre-MDT0001: new disk, initializing [ 2527.324344] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2527.330226] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2529.134381] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2531.952434] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2534.895025] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2534.928426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2535.049274] Lustre: lustre-OST0000: new disk, initializing [ 2535.051845] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2536.200309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2536.206502] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2536.262146] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2537.679802] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2542.834588] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2542.870653] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2544.466421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2544.472589] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2544.521301] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2545.576994] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 2550.949714] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2552.632721] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2789.212900] Lustre: 127710:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 2789.218445] Lustre: 127710:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 2789.223558] Lustre: 127710:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 2789.226850] Lustre: 127710:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 2789.230576] Lustre: 127710:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 2789.234778] Lustre: 127710:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 2942.706153] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 23:28:55 (1764736135) [ 2957.281382] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2957.284351] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2957.290276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2961.295983] LustreError: 127943:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 2961.298624] LustreError: 127943:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2961.593053] Lustre: server umount lustre-MDT0000 complete [ 2964.452286] LustreError: 118941:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2964.502909] LustreError: 118941:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2966.255769] LustreError: 118926:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764736159 with bad export cookie 18114272819006914215 [ 2966.260749] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2966.290990] LustreError: 118926:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2973.041560] Lustre: server umount lustre-MDT0001 complete [ 2983.573590] Lustre: server umount lustre-OST0000 complete [ 2993.852628] Lustre: server umount lustre-OST0001 complete [ 3000.836862] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 3011.888963] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 3020.460954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3025.584264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3030.293993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3035.273304] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3039.945898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3040.002454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3040.156867] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3040.185354] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3040.188269] Lustre: Skipped 1 previous similar message [ 3040.250530] Lustre: lustre-MDT0000: new disk, initializing [ 3040.254347] Lustre: Skipped 1 previous similar message [ 3040.335119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3040.339469] Lustre: Skipped 5 previous similar messages [ 3040.358712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3043.247659] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3051.835452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3051.896443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3051.962473] Lustre: 130751:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3052.193421] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3055.587608] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3061.478563] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3064.014925] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3068.950196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3068.999307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3069.200464] Lustre: lustre-OST0000: new disk, initializing [ 3069.203218] Lustre: Skipped 1 previous similar message [ 3069.208210] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3069.212098] Lustre: Skipped 2 previous similar messages [ 3070.742807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3070.753882] Lustre: Skipped 1 previous similar message [ 3070.767934] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3070.946634] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3073.634616] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 3078.855613] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3085.713290] Lustre: lustre-MDD0000: changelog on [ 3087.667888] Lustre: lustre-MDD0001: changelog on [ 4384.097893] Lustre: lustre-MDD0001: changelog off [ 4385.195312] Lustre: 135031:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 4385.678755] Lustre: lustre-MDD0000: changelog off [ 4391.909917] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4391.910980] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4391.913307] LustreError: Skipped 1 previous similar message [ 4391.919303] Lustre: Skipped 5 previous similar messages [ 4391.921808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4391.921828] Lustre: Skipped 8 previous similar messages [ 4394.352098] LustreError: 140465:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4394.355040] LustreError: 140465:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 4394.420776] Lustre: server umount lustre-OST0000 complete [ 4397.024928] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4397.025816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4397.031045] Lustre: Skipped 2 previous similar messages [ 4402.447265] Lustre: server umount lustre-MDT0000 complete [ 4404.744165] LustreError: 130743:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764737597 with bad export cookie 18114272819007719271 [ 4404.747731] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4404.749194] LustreError: 130743:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4404.970701] Lustre: server umount lustre-MDT0001 complete [ 4411.195749] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 4413.050699] Key type lgssc unregistered [ 4413.226070] LNet: 141400:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4413.230575] LNetError: 141400:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4414.246145] LNet: Removed LNI 192.168.201.148@tcp [ 4414.670237] Key type .llcrypt unregistered [ 4414.672247] Key type ._llcrypt unregistered [ 4424.269959] Key type ._llcrypt registered [ 4424.271975] Key type .llcrypt registered [ 4424.324349] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 4432.357964] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 4432.749218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4432.806782] alg: No test for adler32 (adler32-zlib) [ 4433.696806] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 4433.817126] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 4435.431248] Key type lgssc registered [ 4435.940197] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4440.122629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4443.045745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4445.747865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4448.653595] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4451.768172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4451.791191] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4451.798841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4452.936758] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4452.954559] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4452.996645] Lustre: lustre-MDT0000: new disk, initializing [ 4453.040813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4453.056143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4454.839477] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4459.930861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4459.963972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4460.005540] Lustre: 144550:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4460.031936] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4460.034248] Lustre: Skipped 1 previous similar message [ 4460.072726] Lustre: lustre-MDT0001: new disk, initializing [ 4460.118859] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4460.135187] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4460.139111] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4461.888748] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4466.061031] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4467.766610] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4471.220812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4471.254039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4471.401056] Lustre: lustre-OST0000: new disk, initializing [ 4471.403952] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4471.449108] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4473.113924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4473.118633] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4473.176516] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4474.345663] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4478.455958] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4482.267344] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_module obdecho/obdecho [ 4483.174149] LustreError: 147175:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 4483.178578] LustreError: 147175:0:(obd_config.c:783:class_setup()) setup ec failed (-22) [ 4483.983434] LustreError: 147271:0:(echo_client.c:1880:echo_md_handler()) Can not resolve the path : rc = -2 [ 4488.673979] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4488.674727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4488.679493] Lustre: Skipped 2 previous similar messages [ 4488.682029] Lustre: Skipped 1 previous similar message [ 4493.792835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4493.795274] Lustre: Skipped 2 previous similar messages [ 4494.092511] LustreError: 147611:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4494.137316] Lustre: server umount lustre-MDT0000 complete [ 4495.839750] LustreError: 144541:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764737688 with bad export cookie 10815264489685380950 [ 4495.843470] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4495.845494] LustreError: 144541:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4495.918422] LustreError: 147812:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4495.920684] LustreError: 147812:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4495.993915] Lustre: server umount lustre-MDT0001 complete [ 4506.931589] LustreError: 148012:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4506.933889] LustreError: 148012:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4506.988712] Lustre: server umount lustre-OST0000 complete [ 4515.349197] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 4519.780709] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 4524.656775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4527.817080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4530.926542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4534.318293] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4537.855746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4537.904343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4538.036853] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4538.057818] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4538.100822] Lustre: lustre-MDT0000: new disk, initializing [ 4538.140423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4538.150897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4540.084689] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4545.539997] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4545.570408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4545.612227] Lustre: 150956:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4545.645885] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4545.648633] Lustre: Skipped 1 previous similar message [ 4545.690468] Lustre: lustre-MDT0001: new disk, initializing [ 4545.736670] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4545.754249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4545.758799] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4547.672201] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4551.783818] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4553.450764] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4556.703116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4556.750571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4556.896875] Lustre: lustre-OST0000: new disk, initializing [ 4556.900499] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4556.939264] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4559.587642] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4560.962669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4560.966557] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4561.015423] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4563.493638] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4565.396969] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4565.488417] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4567.193867] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4567.292707] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4571.104842] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4571.108340] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4571.115051] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4571.619380] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4576.739698] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4576.742736] Lustre: Skipped 2 previous similar messages [ 4581.858969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4581.862411] Lustre: Skipped 1 previous similar message [ 4582.367226] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4582.391844] LustreError: 153895:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4582.395308] LustreError: 153895:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4582.460445] Lustre: server umount lustre-OST0000 complete [ 4586.977103] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4586.977147] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4586.981735] Lustre: Skipped 1 previous similar message [ 4590.485886] LustreError: 154097:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4590.488579] LustreError: 154097:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4590.529719] Lustre: server umount lustre-MDT0000 complete [ 4592.230604] LustreError: 151735:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764737785 with bad export cookie 10815264489685381909 [ 4592.232801] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4592.236452] LustreError: 151735:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4592.389993] Lustre: server umount lustre-MDT0001 complete [ 4594.010912] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 23:56:26 (1764737786) [ 4594.809671] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 4595.636057] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 23:56:28 (1764737788) [ 4598.938185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4599.138178] LustreError: 154971:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4599.147295] LustreError: 154971:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4599.169444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4600.805852] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4604.384327] LustreError: 154971:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4604.395285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4606.165723] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4608.501144] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4610.076786] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4612.913870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4615.423413] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4617.824268] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4631.311625] LustreError: 157240:0:(ldlm_resource.c:981:ldlm_resource_complain()) mdt-lustre-MDT0000_UUID: namespace resource [0x200000bd1:0x2f:0x0].0x0 (000000002812c209) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 4633.367511] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.48@tcp (stopping) [ 4633.370725] Lustre: Skipped 2 previous similar messages [ 4634.594336] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4634.598477] Lustre: Skipped 2 previous similar messages [ 4637.449844] LustreError: 157240:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4637.453376] LustreError: 157240:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 4637.482395] Lustre: server umount lustre-MDT0000 complete [ 4638.480936] LustreError: 154966:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4639.063114] LustreError: 154950:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764737832 with bad export cookie 10815264489685382791 [ 4639.066345] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4639.067870] LustreError: 154950:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4639.712208] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4639.717253] Lustre: Skipped 1 previous similar message [ 4643.598799] LustreError: 156180:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4643.605946] LustreError: 156180:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4645.258152] Lustre: server umount lustre-MDT0001 complete [ 4647.929433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4653.297830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4653.423490] LustreError: 158500:0:(llog.c:1643:llog_backup()) MGC192.168.201.148@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 4653.427679] Lustre: 158500:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.201.148@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 4664.078553] LustreError: 158508:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4664.086349] LustreError: 158508:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4674.318580] LustreError: 158508:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4674.325652] LustreError: 158508:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4694.030888] LustreError: 156578:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4694.039939] LustreError: 156578:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [ 4714.463458] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 4714.466086] Lustre: 158500:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.201.148@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 4729.615331] LustreError: 158506:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4729.621818] LustreError: 158506:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 19 previous similar messages [ 4741.599362] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 4741.604391] Lustre: 154962:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.201.148@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 4754.919451] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4754.921695] Lustre: Skipped 2 previous similar messages [ 4756.571206] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4759.012605] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4760.424152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4761.631407] LustreError: 154962:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 4784.613411] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x961789c06b980fee [ 4784.617514] Lustre: MGC192.168.201.148@tcp: Connection restored to 0@lo (at 0@lo) [ 4786.473956] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4788.857320] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4789.603292] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4789.735054] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4791.244762] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4791.987065] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4799.968472] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4799.969544] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4799.971080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4799.971087] Lustre: Skipped 6 previous similar messages [ 4799.974869] Lustre: Skipped 1 previous similar message [ 4803.108876] LustreError: 159984:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4803.111189] LustreError: 159984:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 4803.139337] Lustre: server umount lustre-MDT0000 complete [ 4804.593394] LustreError: 154951:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764737997 with bad export cookie 10815264489685389294 [ 4804.596991] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4804.597755] LustreError: 154951:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4804.718556] Lustre: server umount lustre-MDT0001 complete [ 4816.487260] Lustre: server umount lustre-OST0000 complete [ 4824.378781] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 4828.279621] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 4831.961605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4834.369406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4836.699470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4839.079343] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4841.568970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4841.595018] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4841.686040] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4841.699509] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4841.735325] Lustre: lustre-MDT0000: new disk, initializing [ 4841.762094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4841.765229] Lustre: Skipped 1 previous similar message [ 4841.773304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4843.259281] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4847.822727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4847.851355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4847.889463] Lustre: 163331:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4848.001380] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4849.471406] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4852.967207] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4854.446815] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4857.146968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4857.176686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4857.918271] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4857.962922] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4859.664633] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4863.474755] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4865.311360] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4865.421047] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4867.236854] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4867.351586] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4868.576151] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4868.579035] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4868.582881] Lustre: Skipped 1 previous similar message [ 4868.584880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4868.586966] Lustre: Skipped 2 previous similar messages [ 4882.399120] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4882.423565] LustreError: 166270:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4882.426556] LustreError: 166270:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4882.474093] Lustre: server umount lustre-OST0000 complete [ 4890.222534] Lustre: server umount lustre-MDT0000 complete [ 4891.905190] LustreError: 163323:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764738085 with bad export cookie 10815264489685406080 [ 4891.907634] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4891.909771] LustreError: 163323:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4892.039982] Lustre: server umount lustre-MDT0001 complete [ 4895.898485] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 00:01:28 (1764738088) [ 4898.512321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4898.664206] LustreError: 167359:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4898.671630] LustreError: 167359:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 41 previous similar messages [ 4900.243163] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4903.559229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4905.500859] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4907.933688] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4909.561625] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4912.766798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4915.470956] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 4918.010952] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4928.370862] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 00:02:01 (1764738121) [ 4933.600697] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4933.601086] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4933.602750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4933.602756] Lustre: Skipped 6 previous similar messages [ 4933.604169] LustreError: Skipped 1 previous similar message [ 4933.616017] Lustre: Skipped 4 previous similar messages [ 4935.641370] Lustre: server umount lustre-OST0000 complete [ 4943.258944] Lustre: server umount lustre-MDT0000 complete [ 4944.870450] LustreError: 167338:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764738138 with bad export cookie 10815264489685406948 [ 4944.874300] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4944.875102] LustreError: 167338:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4945.014332] Lustre: server umount lustre-MDT0001 complete [ 4949.989404] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 4951.237286] Key type lgssc unregistered [ 4951.401572] LNet: 170736:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4951.404867] LNetError: 170736:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4951.412399] LNet: Removed LNI 192.168.201.148@tcp [ 4951.822412] Key type .llcrypt unregistered [ 4951.824219] Key type ._llcrypt unregistered [ 4961.796155] Key type ._llcrypt registered [ 4961.797783] Key type .llcrypt registered [ 4961.861967] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 4962.254198] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4962.292715] alg: No test for adler32 (adler32-zlib) [ 4963.197661] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 4963.316659] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 4964.927421] Key type lgssc registered [ 4965.442777] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4969.033721] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4969.043479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4984.543509] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4989.663371] LustreError: 171937:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.148@tcp: failed processing log, type 4: rc = -110 [ 4994.783584] LustreError: 171937:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 4994.791923] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 4994.795616] LustreError: MGC192.168.201.148@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 4994.800204] LustreError: 171937:0:(tgt_mount.c:1729:server_start_targets()) failed to start server lustre-OST0000: -22 [ 4994.804449] LustreError: 171937:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -22 [ 4994.807903] LustreError: 171937:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 4994.810670] LustreError: 171937:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 4994.850141] Lustre: server umount lustre-OST0000 complete [ 4994.852674] LustreError: 171937:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 4999.403925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5000.636325] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5002.403708] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 5006.309498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5006.501586] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5008.183903] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 5010.291526] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5011.776705] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5014.451505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5014.533077] LustreError: 174029:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 5014.537571] Lustre: 174029:0:(mgc_request_server.c:712:mgc_llog_local_copy()) MGC192.168.201.148@tcp: invalid local config log lustre-OST0000: rc = -22 [ 5014.594951] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5016.788541] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 5019.190668] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5022.178039] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5022.178665] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5022.179564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5022.183762] Lustre: Skipped 1 previous similar message [ 5026.259135] LustreError: 174675:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5026.262838] LustreError: 174675:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 5026.311512] Lustre: server umount lustre-MDT0000 complete [ 5027.297655] LustreError: 173600:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5027.305401] LustreError: 173600:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5028.164910] LustreError: 172419:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764738221 with bad export cookie 8786913621026441349 [ 5028.165585] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5028.170023] LustreError: 172419:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5028.267056] LustreError: 174880:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5028.270295] LustreError: 174880:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5028.359302] Lustre: server umount lustre-MDT0001 complete [ 5040.627963] LustreError: 175081:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5040.630642] LustreError: 175081:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5040.689038] Lustre: server umount lustre-OST0000 complete [ 5044.460063] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 5048.729320] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 5053.001843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5055.843243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5058.608561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5061.668399] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5064.755038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5064.787303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5064.897621] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5064.918228] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5064.958480] Lustre: lustre-MDT0000: new disk, initializing [ 5064.989229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5064.997967] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5066.749355] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 5072.045806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5072.074574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5072.116967] Lustre: 177413:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5072.147122] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5072.149342] Lustre: Skipped 1 previous similar message [ 5072.189330] Lustre: lustre-MDT0001: new disk, initializing [ 5072.231982] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5072.247131] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5072.253167] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5073.841175] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 5077.495666] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5079.091200] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5081.695226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5081.724957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5081.834719] Lustre: lustre-OST0000: new disk, initializing [ 5081.837514] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5081.872622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5084.216763] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 5087.951576] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5089.293350] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5089.297937] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5089.356269] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5089.730695] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5089.827827] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5091.623603] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5091.732914] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5094.372454] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5094.372901] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5094.376296] LustreError: Skipped 1 previous similar message [ 5094.381286] Lustre: Skipped 1 previous similar message [ 5094.382559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5094.389019] Lustre: Skipped 3 previous similar messages [ 5098.607243] LustreError: 180351:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5098.610070] LustreError: 180351:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5098.664104] Lustre: server umount lustre-OST0000 complete [ 5103.071946] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5103.074891] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5103.079194] Lustre: Skipped 1 previous similar message [ 5103.081581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5106.268872] Lustre: server umount lustre-MDT0000 complete [ 5107.705130] LustreError: 178193:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764738300 with bad export cookie 8786913621026442049 [ 5107.708881] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5107.710296] LustreError: 178193:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5107.775769] LustreError: 180753:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5107.778748] LustreError: 180753:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 5107.834512] Lustre: server umount lustre-MDT0001 complete [ 5111.344808] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 00:05:04 (1764738304) [ 5118.649610] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 5119.762149] Key type lgssc unregistered [ 5119.899423] LNet: 181936:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5119.902768] LNetError: 181936:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5119.910300] LNet: Removed LNI 192.168.201.148@tcp [ 5120.230328] Key type .llcrypt unregistered [ 5120.231752] Key type ._llcrypt unregistered [ 5128.616929] Key type ._llcrypt registered [ 5128.618560] Key type .llcrypt registered [ 5128.667572] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 5129.127870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5129.136679] alg: No test for adler32 (adler32-zlib) [ 5130.020535] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 5130.120447] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 5131.719581] Key type lgssc registered [ 5132.200962] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5135.421287] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5135.433444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5136.641585] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5138.135603] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 5141.726438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5141.880737] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5143.441779] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 5145.519650] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5146.968973] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5149.868975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5149.943414] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5149.944868] LustreError: 184749:0:(llog.c:1684:llog_backup()) MGC192.168.201.148@tcp: failed to backup log lustre-OST0000: rc = -28 [ 5149.947883] Lustre: 184749:0:(mgc_request_server.c:709:mgc_llog_local_copy()) MGC192.168.201.148@tcp: can't backup local config lustre-OST0000: rc = -28 [ 5149.985149] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5152.105924] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 5154.448751] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5155.338141] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5155.340387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5155.345395] Lustre: Skipped 2 previous similar messages [ 5155.348474] Lustre: Skipped 1 previous similar message [ 5160.417409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5160.420354] Lustre: Skipped 2 previous similar messages [ 5161.471190] LustreError: 185395:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5161.498373] Lustre: server umount lustre-MDT0000 complete [ 5162.970502] LustreError: 183093:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764738356 with bad export cookie 13472767954878125070 [ 5162.973695] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5162.975598] LustreError: 183093:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5163.048785] LustreError: 185600:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5163.052096] LustreError: 185600:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5163.115241] Lustre: server umount lustre-MDT0001 complete [ 5173.874599] LustreError: 185800:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5173.877645] LustreError: 185800:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5173.921218] Lustre: server umount lustre-OST0000 complete [ 5177.376934] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 5181.054429] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 5185.108577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5187.717494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5190.261724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5193.097129] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5195.943417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5195.971569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5196.065347] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5196.082201] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5196.122133] Lustre: lustre-MDT0000: new disk, initializing [ 5196.152915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5196.161216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5197.693802] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 5202.530620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5202.555448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5202.591984] Lustre: 188134:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5202.613828] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5202.615276] Lustre: Skipped 1 previous similar message [ 5202.616776] LustreError: 188900:0:(llog.c:1684:llog_backup()) MGC192.168.201.148@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 5202.619656] LustreError: 188900:0:(llog.c:1684:llog_backup()) Skipped 1 previous similar message [ 5202.622380] Lustre: 188900:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.201.148@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 5202.633325] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5202.635604] Lustre: Skipped 1 previous similar message [ 5202.675821] Lustre: lustre-MDT0001: new disk, initializing [ 5202.716929] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5202.731392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5202.736190] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5204.201059] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 5207.663541] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5209.135901] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5212.056728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5212.089616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5212.221636] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5212.223338] Lustre: Skipped 1 previous similar message [ 5212.225492] LustreError: 189853:0:(llog.c:1684:llog_backup()) MGC192.168.201.148@tcp: failed to backup log lustre-OST0000: rc = -28 [ 5212.229285] LustreError: 189863:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5212.230131] LustreError: 189853:0:(llog.c:1684:llog_backup()) Skipped 1 previous similar message [ 5212.241689] LustreError: 189863:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5212.249354] Lustre: 189853:0:(mgc_request_server.c:732:mgc_llog_local_copy()) MGC192.168.201.148@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 5212.255081] Lustre: 189853:0:(mgc_request_server.c:732:mgc_llog_local_copy()) Skipped 1 previous similar message [ 5212.267460] Lustre: lustre-OST0000: new disk, initializing [ 5212.270134] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5212.315936] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5214.756805] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 5217.787757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5217.792260] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5217.803662] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 5218.529627] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5220.241350] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5220.339787] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5222.075372] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5222.169760] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5228.004208] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5228.004427] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5228.016459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5228.018330] Lustre: Skipped 1 previous similar message [ 5228.024505] Lustre: Skipped 1 previous similar message [ 5233.122286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5233.124774] Lustre: Skipped 1 previous similar message [ 5237.215424] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5237.245099] LustreError: 191074:0:(obd_class.h:479:obd_check_dev()) Device 20 not setup [ 5237.248781] LustreError: 191074:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5237.307569] Lustre: server umount lustre-OST0000 complete [ 5243.360485] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5243.362949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5243.366256] Lustre: Skipped 1 previous similar message [ 5245.196386] LustreError: 191278:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 5245.199346] LustreError: 191278:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5245.246320] Lustre: server umount lustre-MDT0000 complete [ 5246.909831] LustreError: 189880:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764738440 with bad export cookie 13472767954878125756 [ 5246.912309] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5246.914943] LustreError: 189880:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5247.062352] Lustre: server umount lustre-MDT0001 complete [ 5250.858946] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 00:07:23 (1764738443) [ 5257.763894] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 5262.193471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5262.399034] LustreError: 192847:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5262.425053] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5264.080358] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 5267.424252] LustreError: 192847:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5267.599706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5269.426887] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 5270.577862] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5273.318038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5275.848918] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 5279.384059] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5279.417315] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5279.491558] Lustre: lustre-OST0001: new disk, initializing [ 5279.494560] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5281.852282] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 5282.126452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5282.130975] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5282.176680] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5288.081308] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5293.258273] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5299.039685] Lustre: DEBUG MARKER: ADD OST3 [ 5300.933803] loop: module loaded [ 5300.956459] loop0: detected capacity change from 0 to 204800000 [ 5301.047107] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5301.052240] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5301.057519] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5301.112542] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5303.355700] loop0: detected capacity change from 0 to 204800000 [ 5305.551826] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5305.581810] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5305.656216] Lustre: lustre-OST0003: new disk, initializing [ 5305.659598] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 5305.710315] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 5305.713818] Lustre: Skipped 3 previous similar messages [ 5306.256934] Lustre: DEBUG MARKER: STOP OST3 [ 5306.835464] Lustre: *** cfs_fail_loc=2109, val=2*** [ 5306.880666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 5306.885824] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 5306.900576] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 5307.078116] Lustre: Failing over lustre-OST0003 [ 5307.106841] LustreError: 197150:0:(obd_class.h:479:obd_check_dev()) Device 31 not setup [ 5307.109372] LustreError: 197150:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5307.136144] Lustre: server umount lustre-OST0003 complete [ 5307.295422] LustreError: 194299:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.201.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5308.170491] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 5308.896210] LustreError: 197083:0:(osp_precreate.c:1280:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 5311.968356] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5311.972288] LustreError: Skipped 1 previous similar message [ 5311.974593] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5311.978981] LustreError: 194865:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5311.979476] Lustre: Skipped 1 previous similar message [ 5311.985373] LustreError: 194865:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5312.031809] Lustre: DEBUG MARKER: START OST3 again [ 5314.689626] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5314.793333] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 5316.526820] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5316.535896] Lustre: lustre-OST0003: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5316.536410] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5316.956376] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 5330.703185] Lustre: Failing over lustre-OST0003 [ 5330.726105] LustreError: 199140:0:(obd_class.h:479:obd_check_dev()) Device 31 not setup [ 5330.728189] LustreError: 199140:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5330.746944] Lustre: server umount lustre-OST0003 complete [ 5331.936513] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5331.936633] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5331.940205] LustreError: Skipped 1 previous similar message [ 5331.941217] LustreError: 194865:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5331.945112] Lustre: Skipped 1 previous similar message [ 5331.946778] LustreError: 194865:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5344.736388] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5344.739076] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5344.744386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5344.746135] Lustre: Skipped 1 previous similar message [ 5349.597436] Lustre: server umount lustre-MDT0000 complete [ 5350.881363] LustreError: 198885:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5350.887191] LustreError: 198885:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 5350.896514] LustreError: 195111:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764738544 with bad export cookie 13472767954878126624 [ 5350.900431] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5350.901221] LustreError: 195111:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5361.121234] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5361.124202] Lustre: Skipped 7 previous similar messages [ 5365.215238] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5365.260744] LustreError: 199610:0:(obd_class.h:479:obd_check_dev()) Device 33 not setup [ 5365.263303] LustreError: 199610:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5365.333861] Lustre: server umount lustre-MDT0001 complete [ 5366.886634] Lustre: server umount lustre-OST0000 complete [ 5368.468263] Lustre: server umount lustre-OST0001 complete [ 5370.775454] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 5373.644302] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 5377.364344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5379.729629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5381.843537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5384.094193] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5386.529046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5386.557996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5386.648342] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5386.663639] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5386.699315] Lustre: lustre-MDT0000: new disk, initializing [ 5386.725407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5386.728758] Lustre: Skipped 1 previous similar message [ 5386.736137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5388.117554] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 5392.457089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5392.483171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5392.516831] Lustre: 202193:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5392.624653] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5394.011202] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 5397.073768] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5398.336564] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5400.766993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5400.792575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5402.884547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5402.888164] Lustre: Skipped 1 previous similar message [ 5402.890633] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5402.929995] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5403.044481] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 5406.521984] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5408.005111] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5408.091987] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5409.505278] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5409.592699] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5413.346518] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5413.347922] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5413.349292] LustreError: Skipped 1 previous similar message [ 5413.352796] Lustre: Skipped 5 previous similar messages [ 5413.353452] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5413.357444] Lustre: Skipped 1 previous similar message [ 5416.418017] Lustre: server umount lustre-OST0000 complete [ 5432.287195] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5432.330685] LustreError: 205334:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5432.333286] LustreError: 205334:0:(obd_class.h:479:obd_check_dev()) Skipped 23 previous similar messages [ 5432.361656] Lustre: server umount lustre-MDT0000 complete [ 5433.739383] LustreError: 202186:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764738626 with bad export cookie 13472767954878128899 [ 5433.742153] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5433.744088] LustreError: 202186:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5433.876546] Lustre: server umount lustre-MDT0001 complete [ 5435.128881] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 00:10:28 (1764738628) [ 5441.057227] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 5444.565741] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 5448.194081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5450.499079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5452.720556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5455.197217] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5457.584034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5457.606166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5457.698629] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5457.711588] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5457.713505] Lustre: Skipped 3 previous similar messages [ 5457.758316] Lustre: lustre-MDT0000: new disk, initializing [ 5457.760249] Lustre: Skipped 2 previous similar messages [ 5457.793828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5459.203107] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 5463.569503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5463.593262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5463.629522] Lustre: 208372:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5463.755499] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5465.136566] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 5468.482089] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5469.877441] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5472.438050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5472.467699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5473.968365] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5474.836035] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 5478.327419] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5479.972483] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5480.062620] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5481.579255] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5481.673423] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5484.515072] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5484.517788] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5484.518208] LustreError: Skipped 1 previous similar message [ 5484.523077] Lustre: Skipped 3 previous similar messages [ 5484.524107] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5484.528398] Lustre: Skipped 7 previous similar messages [ 5488.531892] Lustre: server umount lustre-OST0000 complete [ 5496.030924] Lustre: server umount lustre-MDT0000 complete [ 5497.308581] LustreError: 208363:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764738690 with bad export cookie 13472767954878129781 [ 5497.312369] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5497.312799] LustreError: 208363:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5497.432101] Lustre: server umount lustre-MDT0001 complete [ 5500.362350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5500.527654] LustreError: 212191:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5500.535847] LustreError: 212191:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5501.856920] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 5504.766879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5506.377750] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 5508.171780] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5509.448940] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5511.709182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5513.713123] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 5515.498840] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5535.714835] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5535.717765] LustreError: Skipped 2 previous similar messages [ 5538.454281] Lustre: server umount lustre-OST0000 complete [ 5554.143092] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5554.200171] Lustre: server umount lustre-MDT0000 complete [ 5555.432358] LustreError: 212172:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764738748 with bad export cookie 13472767954878130663 [ 5555.435624] LustreError: 212172:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5555.435786] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5555.551311] Lustre: server umount lustre-MDT0001 complete [ 5648.191195] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 5649.146695] Key type lgssc unregistered [ 5649.274406] LNet: 215526:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5649.277189] LNetError: 215526:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5649.284413] LNet: Removed LNI 192.168.201.148@tcp [ 5649.560096] Key type .llcrypt unregistered [ 5649.561257] Key type ._llcrypt unregistered [ 5657.658889] Key type ._llcrypt registered [ 5657.659840] Key type .llcrypt registered [ 5657.694814] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 5663.674184] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 5664.065545] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5664.079560] alg: No test for adler32 (adler32-zlib) [ 5664.938528] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 5665.017904] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 5666.599143] Key type lgssc registered [ 5666.947689] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5669.822837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5671.949216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5674.002759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5676.180194] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5678.462401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5678.482317] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5678.488543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5679.574946] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5679.588601] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5679.628150] Lustre: lustre-MDT0000: new disk, initializing [ 5679.655104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5679.661653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5681.037868] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 5685.292379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5685.317341] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5685.354408] Lustre: 218825:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5685.379480] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5685.382521] Lustre: Skipped 1 previous similar message [ 5685.419231] Lustre: lustre-MDT0001: new disk, initializing [ 5685.457603] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5685.472123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5685.475679] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5686.820831] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 5689.948139] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5691.102675] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5693.433593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5693.455664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5693.544804] Lustre: lustre-OST0000: new disk, initializing [ 5693.546862] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5693.573390] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5694.905659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5694.909364] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5694.941624] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5695.466783] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 5698.540765] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5699.919892] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5699.994556] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5701.371559] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5701.444134] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5705.185312] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5705.186319] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5705.187976] LustreError: Skipped 1 previous similar message [ 5705.188684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5705.192161] Lustre: Skipped 1 previous similar message [ 5708.142084] LustreError: 221763:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5708.178956] Lustre: server umount lustre-OST0000 complete [ 5710.304740] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5710.305241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5710.308448] Lustre: Skipped 1 previous similar message [ 5710.311425] Lustre: Skipped 2 previous similar messages [ 5715.424098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5715.426154] Lustre: Skipped 1 previous similar message [ 5715.518296] LustreError: 221964:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5715.520262] LustreError: 221964:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5715.547211] Lustre: server umount lustre-MDT0000 complete [ 5716.883151] LustreError: 218818:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764738910 with bad export cookie 4157397030195469264 [ 5716.886116] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5716.886442] LustreError: 218818:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5716.946860] LustreError: 222166:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5716.949405] LustreError: 222166:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5716.996124] Lustre: server umount lustre-MDT0001 complete [ 5722.904800] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 5726.393550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5726.557889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5727.886919] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 5730.709763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5730.812540] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5732.154842] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 5733.120116] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5735.355770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5737.266938] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 5740.102336] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5740.118720] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5740.178774] Lustre: lustre-OST0001: new disk, initializing [ 5740.180630] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5740.212914] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5740.214650] Lustre: Skipped 1 previous similar message [ 5742.103593] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 5743.814724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5743.817664] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5743.848881] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5747.543793] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5751.167938] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5759.783175] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 00:15:52 (1764738952) [ 5761.503789] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5761.505811] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5761.509977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5764.577524] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5764.580800] Lustre: Skipped 2 previous similar messages [ 5766.528539] LustreError: 226712:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5766.530727] LustreError: 226712:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5766.558098] Lustre: server umount lustre-MDT0000 complete [ 5767.749162] LustreError: 224056:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764738960 with bad export cookie 4157397030195470132 [ 5767.752651] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5767.753489] LustreError: 224056:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5767.865375] Lustre: server umount lustre-MDT0001 complete [ 5779.891148] LustreError: 227114:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5779.893611] LustreError: 227114:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 5779.911756] Lustre: server umount lustre-OST0000 complete [ 5791.411116] LustreError: 227315:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5791.414233] LustreError: 227315:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5791.467163] Lustre: server umount lustre-OST0001 complete [ 5793.619848] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 5797.191126] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 5800.645798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5803.024929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5805.242070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5807.391298] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5809.481134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5809.500470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5809.573720] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5809.585052] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5809.617834] Lustre: lustre-MDT0000: new disk, initializing [ 5809.640841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5809.647892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5810.817772] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 5814.812126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5814.833795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5814.860174] Lustre: 229498:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5814.959477] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5816.211828] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 5819.199069] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5820.353163] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5822.686841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5822.715774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5822.811744] Lustre: lustre-OST0000: new disk, initializing [ 5822.813093] Lustre: Skipped 1 previous similar message [ 5822.814848] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5822.817281] Lustre: Skipped 2 previous similar messages [ 5824.759158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5824.762024] Lustre: Skipped 1 previous similar message [ 5824.764233] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5824.772992] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 5824.795530] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5827.741557] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5829.052546] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5829.121509] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5830.325622] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5830.394057] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5833.003845] LustreError: 232434:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5833.005517] LustreError: 232434:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5833.038309] Lustre: server umount lustre-OST0000 complete [ 5835.233045] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5835.233647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5835.236542] Lustre: Skipped 1 previous similar message [ 5835.238427] Lustre: Skipped 4 previous similar messages [ 5844.471623] Lustre: server umount lustre-MDT0000 complete [ 5845.471957] LustreError: 229508:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5845.476900] LustreError: 229508:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5845.655682] LustreError: 229491:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764739038 with bad export cookie 4157397030195472232 [ 5845.658899] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5845.660069] LustreError: 229491:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5855.247993] Lustre: server umount lustre-MDT0001 complete [ 5857.965267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5858.108815] LustreError: 233316:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5858.133258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5858.135870] Lustre: Skipped 2 previous similar messages [ 5859.349672] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 5862.025519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5863.371590] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 5865.013679] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5866.153911] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5868.438770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5868.513027] LustreError: 234915:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5870.330170] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 5872.054340] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6015.969706] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6015.973597] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6015.973898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6015.977994] Lustre: Skipped 4 previous similar messages [ 6020.591090] LustreError: 235602:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6020.593360] LustreError: 235602:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 6020.627818] Lustre: server umount lustre-OST0000 complete [ 6026.208614] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6027.988247] Lustre: server umount lustre-MDT0000 complete [ 6029.225210] LustreError: 234035:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764739222 with bad export cookie 4157397030195473114 [ 6029.228388] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6029.231049] LustreError: 234035:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6029.348782] Lustre: server umount lustre-MDT0001 complete [ 6136.628560] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 6137.825330] Key type lgssc unregistered [ 6137.981439] LNet: 236571:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6137.983907] LNetError: 236571:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6137.992410] LNet: Removed LNI 192.168.201.148@tcp [ 6138.341101] Key type .llcrypt unregistered [ 6138.342154] Key type ._llcrypt unregistered [ 6147.070720] Key type ._llcrypt registered [ 6147.072038] Key type .llcrypt registered [ 6147.116604] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 6153.617911] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 6153.908221] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6153.959602] alg: No test for adler32 (adler32-zlib) [ 6154.808693] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 6154.895558] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 6156.479216] Key type lgssc registered [ 6156.878641] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6159.701652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6161.923223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6164.082908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6166.338389] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6172.110679] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 6176.050362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6176.070778] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6176.076322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6177.186643] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6177.199743] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6177.241381] Lustre: lustre-MDT0000: new disk, initializing [ 6177.268030] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6177.275361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6178.734490] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6183.100513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6183.129660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6183.167620] Lustre: 240569:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6183.194573] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6183.196562] Lustre: Skipped 1 previous similar message [ 6183.232423] Lustre: lustre-MDT0001: new disk, initializing [ 6183.272168] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6183.288996] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6183.294736] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6184.713706] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6187.075623] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6189.390713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6189.429750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6189.556279] Lustre: lustre-OST0000: new disk, initializing [ 6189.558437] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6189.591570] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6190.715589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6190.720794] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6190.768334] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6191.661833] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6196.210714] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6196.241713] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6196.304614] Lustre: lustre-OST0001: new disk, initializing [ 6196.306664] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6196.340245] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6197.633821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6197.637053] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6197.680095] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6198.287083] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6203.626820] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6207.170378] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6218.211742] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6218.213809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6218.216623] Lustre: Skipped 3 previous similar messages [ 6218.218263] Lustre: Skipped 2 previous similar messages [ 6221.065560] LustreError: 244092:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 6221.099961] Lustre: server umount lustre-MDT0000 complete [ 6222.249190] LustreError: 240561:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764739415 with bad export cookie 10499516125768954978 [ 6222.252661] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6222.253197] LustreError: 240561:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6222.321654] LustreError: 244293:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 6222.324356] LustreError: 244293:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6222.374623] Lustre: server umount lustre-MDT0001 complete [ 6233.781150] LustreError: 244493:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6233.784122] LustreError: 244493:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6233.809264] Lustre: server umount lustre-OST0000 complete [ 6245.300490] LustreError: 244695:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6245.302592] LustreError: 244695:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6245.344436] Lustre: server umount lustre-OST0001 complete [ 6250.138792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6253.034058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6253.360550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6256.096722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6258.777475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6259.090134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6265.655532] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 6269.556105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6269.561959] Lustre: lustre-MDT0000: reset Object Index mappings [ 6269.750096] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6271.222628] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6274.093653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6274.098630] Lustre: lustre-MDT0001: reset Object Index mappings [ 6275.456279] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6276.361691] Lustre: 249111:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6278.418991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6278.532261] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6278.534560] Lustre: Skipped 1 previous similar message [ 6279.592452] LustreError: 249467:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6279.596820] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6279.600770] LustreError: 249467:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6280.403836] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6283.186888] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6284.342275] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 6285.089855] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6288.096568] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6289.247523] Lustre: 250957:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6299.619894] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6299.621866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6299.623444] Lustre: Skipped 3 previous similar messages [ 6299.626288] Lustre: Skipped 4 previous similar messages [ 6301.375746] LustreError: 251062:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 6301.378880] LustreError: 251062:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6301.411683] Lustre: server umount lustre-MDT0000 complete [ 6302.659226] LustreError: 247984:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764739495 with bad export cookie 10499516125768961754 [ 6302.662801] LustreError: 247984:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6302.662852] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6302.778081] Lustre: server umount lustre-MDT0001 complete [ 6313.906814] LustreError: 251464:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 6313.908973] LustreError: 251464:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 6313.925092] Lustre: server umount lustre-OST0000 complete [ 6325.470553] Lustre: server umount lustre-OST0001 complete [ 6332.158206] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 00:25:25 (1764739525) [ 6337.290834] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 6340.324875] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 6343.382779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6345.401640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6347.299532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6349.391459] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6351.751352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6351.772295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6351.855643] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6351.868599] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6351.905691] Lustre: lustre-MDT0000: new disk, initializing [ 6351.929084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6351.931018] Lustre: Skipped 1 previous similar message [ 6351.937515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6353.258534] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6357.423420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6357.446336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6357.480487] Lustre: 255181:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6357.590984] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6358.956498] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6362.094099] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6363.346853] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6365.693824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6365.718456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6365.828565] Lustre: lustre-OST0000: new disk, initializing [ 6365.831217] Lustre: Skipped 1 previous similar message [ 6365.834363] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6365.837681] Lustre: Skipped 2 previous similar messages [ 6367.676641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6367.679161] Lustre: Skipped 1 previous similar message [ 6367.681238] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6367.720621] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6367.887487] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6371.043047] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6372.320331] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6372.396780] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6373.687455] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6373.762370] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6376.432073] LustreError: 258119:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6376.433917] LustreError: 258119:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 6376.473716] Lustre: server umount lustre-OST0000 complete [ 6377.952723] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6377.952819] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6377.953370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6377.956859] Lustre: Skipped 1 previous similar message [ 6383.072856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6383.074838] Lustre: Skipped 2 previous similar messages [ 6388.192663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6388.194768] Lustre: Skipped 1 previous similar message [ 6392.287109] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6392.349672] Lustre: server umount lustre-MDT0000 complete [ 6393.312955] LustreError: 255966:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6393.317469] LustreError: 255966:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6393.567242] LustreError: 255174:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764739586 with bad export cookie 10499516125768963847 [ 6393.570678] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6393.571402] LustreError: 255174:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6403.092614] Lustre: server umount lustre-MDT0001 complete [ 6408.869857] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 6412.091760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6412.224493] LustreError: 259696:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6412.238246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6412.240008] Lustre: Skipped 2 previous similar messages [ 6413.510463] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6416.452605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6417.954532] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6418.966300] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6421.279709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6423.372416] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6426.309352] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6426.340813] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6426.404156] Lustre: lustre-OST0001: new disk, initializing [ 6426.405975] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6427.843646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6427.846903] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6427.884503] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6428.502825] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6433.068248] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6434.371818] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6441.032214] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 6441.033790] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 6443.490175] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6443.492484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6443.493609] Lustre: Skipped 3 previous similar messages [ 6443.495283] Lustre: Skipped 2 previous similar messages [ 6448.649335] LustreError: 263202:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 6448.651771] LustreError: 263202:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 6448.684584] Lustre: server umount lustre-MDT0000 complete [ 6449.983322] LustreError: 259677:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764739643 with bad export cookie 10499516125768964729 [ 6449.987349] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6449.987611] LustreError: 259677:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6450.103797] Lustre: server umount lustre-MDT0001 complete [ 6460.930286] Lustre: server umount lustre-OST0000 complete [ 6472.480202] Lustre: server umount lustre-OST0001 complete [ 6477.845310] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 6481.325877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6481.475370] LustreError: 264980:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6481.495133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6481.497133] Lustre: Skipped 3 previous similar messages [ 6482.861058] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6485.808962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6487.444084] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6488.468320] Lustre: 266088:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 6490.751981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6491.937387] LustreError: 266631:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6491.943930] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 6492.947267] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6495.821835] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6496.940515] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 6497.847345] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6501.018772] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6502.267616] Lustre: 267943:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 6509.011974] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 6509.023645] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 6511.980099] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 00:28:25 (1764739705) [ 6526.943992] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6526.946150] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6526.950323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6526.953200] Lustre: Skipped 4 previous similar messages [ 6529.601282] LustreError: 268649:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 6529.603304] LustreError: 268649:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 6529.636700] Lustre: server umount lustre-MDT0000 complete [ 6530.907465] LustreError: 265707:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764739724 with bad export cookie 10499516125768971001 [ 6530.914591] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6530.916332] LustreError: 265707:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6533.088308] LustreError: 266465:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6533.097205] LustreError: 266465:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6545.375058] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6545.459651] Lustre: server umount lustre-MDT0001 complete [ 6561.247182] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6561.279727] Lustre: server umount lustre-OST0000 complete [ 6568.595966] Lustre: server umount lustre-OST0001 complete [ 6570.817539] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 6574.073966] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 6577.697309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6579.803505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6581.919830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6584.121893] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6589.735693] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 6593.266727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6593.291610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6593.374912] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6593.388508] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6593.428222] Lustre: lustre-MDT0000: new disk, initializing [ 6593.458248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6594.735478] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6598.974815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6598.998731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6599.039317] Lustre: 272134:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6599.172951] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6600.504971] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6604.993474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6605.024104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6605.913352] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6607.065739] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6611.414474] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6611.436831] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6611.531565] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6611.534994] Lustre: Skipped 6 previous similar messages [ 6612.711256] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6613.489760] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6618.107128] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6628.500715] Lustre: DEBUG MARKER: == conf-sanity test 160: MGC updates failnodes from all participants ========================================================== 00:30:21 (1764739821) [ 6629.855713] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6629.857564] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6629.860372] Lustre: Skipped 5 previous similar messages [ 6629.862558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6629.864744] Lustre: Skipped 12 previous similar messages [ 6635.355938] Lustre: server umount lustre-MDT0000 complete [ 6636.648596] LustreError: 272125:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764739829 with bad export cookie 10499516125768978855 [ 6636.652615] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6636.652970] LustreError: 272125:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6636.807908] Lustre: server umount lustre-MDT0001 complete [ 6648.769684] Lustre: server umount lustre-OST0000 complete [ 6660.211382] LustreError: 276351:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6660.213387] LustreError: 276351:0:(obd_class.h:479:obd_check_dev()) Skipped 45 previous similar messages [ 6660.263080] Lustre: server umount lustre-OST0001 complete [ 6666.229207] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 6669.649654] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 6672.761022] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6674.933948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6677.061386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6679.457102] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6682.272473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6682.297520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6682.388246] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6682.392347] Lustre: Skipped 2 previous similar messages [ 6682.404768] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6682.406937] Lustre: Skipped 4 previous similar messages [ 6682.446736] Lustre: lustre-MDT0000: new disk, initializing [ 6682.449208] Lustre: Skipped 3 previous similar messages [ 6682.486472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6682.489629] Lustre: Skipped 3 previous similar messages [ 6683.981218] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6688.279536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6688.300581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6688.337917] Lustre: 279189:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6688.453446] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6688.458511] Lustre: Skipped 2 previous similar messages [ 6689.932612] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6693.210499] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6694.523883] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6696.736132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6696.761183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6698.548285] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6698.881571] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6701.920491] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6702.662464] Lustre: Mounted lustre-client [ 6702.978919] LustreError: 281695:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff900a4a07a000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 6703.004411] Lustre: Unmounted lustre-client [ 6707.681711] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6707.683386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6707.686026] Lustre: Skipped 4 previous similar messages [ 6707.689208] Lustre: Skipped 5 previous similar messages [ 6712.149390] Lustre: server umount lustre-OST0000 complete [ 6714.335992] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6719.647618] Lustre: server umount lustre-MDT0000 complete [ 6720.974278] LustreError: 279180:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764739914 with bad export cookie 10499516125768980381 [ 6720.977129] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6720.978183] LustreError: 279180:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6721.100518] Lustre: server umount lustre-MDT0001 complete [ 6724.964553] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 6726.038863] Key type lgssc unregistered [ 6758.943362] LNet: 282972:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6758.946117] LNetError: 282972:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6833.701348] LNet: Removed LNI 192.168.201.148@tcp [ 6834.044159] Key type .llcrypt unregistered [ 6834.045839] Key type ._llcrypt unregistered [ 6840.175750] Key type ._llcrypt registered [ 6840.176872] Key type .llcrypt registered [ 6840.217361] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 6846.790942] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 6847.123208] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6847.156204] alg: No test for adler32 (adler32-zlib) [ 6848.024446] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 6848.110418] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 6849.695255] Key type lgssc registered [ 6850.146932] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6853.232026] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6855.617832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6857.863414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6860.233214] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6860.872769] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 00:34:13 (1764740053) [ 6863.103449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6863.126245] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6863.130659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6864.213941] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6864.225810] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6864.260257] Lustre: lustre-MDT0000: new disk, initializing [ 6864.282860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6864.288548] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6865.606298] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6869.791080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6869.808167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6869.848874] Lustre: 286141:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6869.873731] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6869.875421] Lustre: Skipped 1 previous similar message [ 6869.909767] Lustre: lustre-MDT0001: new disk, initializing [ 6869.941788] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6869.954831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6869.959464] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6871.220803] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6874.256189] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6875.420821] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6877.647621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6877.671874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6877.771670] Lustre: lustre-OST0000: new disk, initializing [ 6877.773725] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6877.802306] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6879.743576] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_default_debug -1 all [ 6882.803495] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6884.857680] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6884.862465] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6884.889290] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6895.074592] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6895.074770] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6895.078315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6895.079866] LustreError: Skipped 1 previous similar message [ 6895.085823] Lustre: Skipped 1 previous similar message [ 6898.093114] LustreError: 288585:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6898.131491] Lustre: server umount lustre-OST0000 complete [ 6900.192794] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6900.193505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6900.199311] Lustre: Skipped 2 previous similar messages [ 6905.312176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6905.313880] Lustre: Skipped 2 previous similar messages [ 6905.473149] LustreError: 288788:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6905.474912] LustreError: 288788:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6905.503811] Lustre: server umount lustre-MDT0000 complete [ 6906.823075] LustreError: 288498:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764740099 with bad export cookie 10484499040129197901 [ 6906.825556] LustreError: MGC192.168.201.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6906.826366] LustreError: 288498:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6906.896820] LustreError: 288990:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6906.899044] LustreError: 288990:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6906.967193] Lustre: server umount lustre-MDT0001 complete [ 6910.588635] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 6911.623733] Key type lgssc unregistered [ 6911.742428] LNet: 289522:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6911.744743] LNetError: 289522:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6911.754378] LNet: Removed LNI 192.168.201.148@tcp [ 6912.112513] Key type .llcrypt unregistered [ 6912.113572] Key type ._llcrypt unregistered [ 6947.095590] Key type ._llcrypt registered [ 6947.097366] Key type .llcrypt registered [ 6947.140275] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 6947.251526] Key type .llcrypt unregistered [ 6947.253205] Key type ._llcrypt unregistered [ 6975.385455] Key type ._llcrypt registered [ 6975.386860] Key type .llcrypt registered [ 6975.424145] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 6975.514420] Key type .llcrypt unregistered [ 6975.515444] Key type ._llcrypt unregistered [ 6986.760725] Key type ._llcrypt registered [ 6986.761815] Key type .llcrypt registered [ 6986.802401] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 6986.903489] Key type .llcrypt unregistered [ 6986.904956] Key type ._llcrypt unregistered [ 6998.123736] Key type ._llcrypt registered [ 6998.124781] Key type .llcrypt registered [ 6998.167692] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 6998.276290] Key type .llcrypt unregistered [ 6998.277367] Key type ._llcrypt unregistered [ 7025.786147] Key type ._llcrypt registered [ 7025.787271] Key type .llcrypt registered [ 7025.828439] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 7025.935516] Key type .llcrypt unregistered [ 7025.936831] Key type ._llcrypt unregistered [ 7036.940145] Key type ._llcrypt registered [ 7036.941312] Key type .llcrypt registered [ 7036.982427] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 7037.084624] Key type .llcrypt unregistered [ 7037.085622] Key type ._llcrypt unregistered [ 7048.565777] Key type ._llcrypt registered [ 7048.566841] Key type .llcrypt registered [ 7048.607520] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 7048.719374] Key type .llcrypt unregistered [ 7048.721138] Key type ._llcrypt unregistered [ 7060.260434] Key type ._llcrypt registered [ 7060.261893] Key type .llcrypt registered [ 7060.303133] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 7060.423459] Key type .llcrypt unregistered [ 7060.424592] Key type ._llcrypt unregistered [ 7071.279277] Key type ._llcrypt registered [ 7071.281210] Key type .llcrypt registered [ 7071.326347] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 7071.431550] Key type .llcrypt unregistered [ 7071.433289] Key type ._llcrypt unregistered [ 7082.536029] Key type ._llcrypt registered [ 7082.537105] Key type .llcrypt registered [ 7082.580712] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 7082.693488] Key type .llcrypt unregistered [ 7082.695205] Key type ._llcrypt unregistered [ 7104.220261] Key type ._llcrypt registered [ 7104.221639] Key type .llcrypt registered [ 7104.255902] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 7104.350392] Key type .llcrypt unregistered [ 7104.351625] Key type ._llcrypt unregistered [ 7115.319073] Key type ._llcrypt registered [ 7115.320081] Key type .llcrypt registered [ 7115.357885] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 7115.458366] Key type .llcrypt unregistered [ 7115.459602] Key type ._llcrypt unregistered [ 7126.561689] Key type ._llcrypt registered [ 7126.562749] Key type .llcrypt registered [ 7126.599211] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 7126.688686] Key type .llcrypt unregistered [ 7126.689973] Key type ._llcrypt unregistered [ 7146.253313] Key type ._llcrypt registered [ 7146.254187] Key type .llcrypt registered [ 7146.299160] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing set_hostid [ 7152.455600] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing load_modules_local [ 7152.868520] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7152.894878] alg: No test for adler32 (adler32-zlib) [ 7153.749534] Lustre: Lustre: Build Version: 2.17.0_RC1_3_g509e2bd [ 7153.831036] LNet: Added LNI 192.168.201.148@tcp [8/256/0/180] [ 7155.415258] Key type lgssc registered [ 7155.811420] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7158.778124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7161.134104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7163.287216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7165.613354] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7166.214234] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 6882 sec ========== 00:39:19 (1764740359) [ 7166.802035] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 00:39:19 (1764740359) === [ 7167.995696] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 00:39:21 (1764740361) === [ 7181.916768] Lustre: DEBUG MARKER: oleg148-server.virtnet: executing unload_modules_local [ 7182.952103] Key type lgssc unregistered [ 7183.070367] LNet: 311215:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7183.072653] LNetError: 311215:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7183.083302] LNet: Removed LNI 192.168.201.148@tcp [ 7183.355466] Key type .llcrypt unregistered [ 7183.356555] Key type ._llcrypt unregistered