[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 405943488 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002155] x2apic enabled [ 0.002968] Switched APIC routing to physical x2apic. [ 0.003006] kvm-guest: setup PV IPIs [ 0.005886] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.006010] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007005] pid_max: default: 32768 minimum: 301 [ 0.008082] LSM: Security Framework initializing [ 0.009025] Yama: becoming mindful. [ 0.010019] SELinux: Initializing. [ 0.011027] *** VALIDATE selinux *** [ 0.017680] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.021600] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.022089] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.023071] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.024063] *** VALIDATE tmpfs *** [ 0.025375] *** VALIDATE proc *** [ 0.026176] *** VALIDATE cgroup *** [ 0.027008] *** VALIDATE cgroup2 *** [ 0.028056] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.029101] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.030003] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.031017] Spectre V2 : User space: Vulnerable [ 0.032003] Speculative Store Bypass: Vulnerable [ 0.034928] debug: unmapping init [mem 0xffffffff96459000-0xffffffff96460fff] [ 0.036145] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.037475] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.038009] ... version: 2 [ 0.038754] ... bit width: 48 [ 0.039015] ... generic registers: 4 [ 0.039781] ... value mask: 0000ffffffffffff [ 0.040005] ... max period: 00007fffffffffff [ 0.041005] ... fixed-purpose events: 3 [ 0.041792] ... event mask: 000000070000000f [ 0.042105] rcu: Hierarchical SRCU implementation. [ 0.044254] smp: Bringing up secondary CPUs ... [ 0.045346] x86: Booting SMP configuration: [ 0.046011] .... node #0, CPUs: #1 #2 #3 [ 0.048487] smp: Brought up 1 node, 4 CPUs [ 0.049751] smpboot: Max logical packages: 1 [ 0.050005] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.259185] node 0 deferred pages initialised in 208ms [ 0.262155] devtmpfs: initialized [ 0.263023] x86/mm: Memory block size: 128MB [ 0.265142] gcov: version magic: 0x41383552 [ 0.267155] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.269048] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.270211] pinctrl core: initialized pinctrl subsystem [ 0.271080] [ 0.271391] ************************************************************* [ 0.273006] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.274005] ** ** [ 0.276006] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.277005] ** ** [ 0.278005] ** This means that this kernel is built to expose internal ** [ 0.279006] ** IOMMU data structures, which may compromise security on ** [ 0.281006] ** your system. ** [ 0.282005] ** ** [ 0.283006] ** If you see this message and you are not debugging the ** [ 0.285005] ** kernel, report this immediately to your vendor! ** [ 0.286005] ** ** [ 0.287006] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.288006] ************************************************************* [ 0.290433] NET: Registered protocol family 16 [ 0.291287] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.292027] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.294027] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.296058] cpuidle: using governor menu [ 0.297391] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.298374] PCI: Using configuration type 1 for base access [ 0.300102] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.305157] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.306022] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.308095] cryptd: max_cpu_qlen set to 1000 [ 0.310054] ACPI: Added _OSI(Module Device) [ 0.310832] ACPI: Added _OSI(Processor Device) [ 0.311007] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.311786] ACPI: Added _OSI(Processor Aggregator Device) [ 0.315138] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.319371] ACPI: Interpreter enabled [ 0.320033] ACPI: PM: (supports S0 S3 S4 S5) [ 0.320757] ACPI: Using IOAPIC for interrupt routing [ 0.322037] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.323206] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.330152] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.332016] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.333007] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.335034] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.337869] acpiphp: Slot [2] registered [ 0.338067] acpiphp: Slot [5] registered [ 0.338881] acpiphp: Slot [6] registered [ 0.340065] acpiphp: Slot [7] registered [ 0.340943] acpiphp: Slot [8] registered [ 0.342060] acpiphp: Slot [9] registered [ 0.342860] acpiphp: Slot [10] registered [ 0.343048] acpiphp: Slot [3] registered [ 0.343813] acpiphp: Slot [4] registered [ 0.345043] acpiphp: Slot [11] registered [ 0.345810] acpiphp: Slot [12] registered [ 0.346040] acpiphp: Slot [13] registered [ 0.346808] acpiphp: Slot [14] registered [ 0.348060] acpiphp: Slot [15] registered [ 0.348840] acpiphp: Slot [16] registered [ 0.349042] acpiphp: Slot [17] registered [ 0.349768] acpiphp: Slot [18] registered [ 0.351043] acpiphp: Slot [19] registered [ 0.351937] acpiphp: Slot [20] registered [ 0.352058] acpiphp: Slot [21] registered [ 0.352985] acpiphp: Slot [22] registered [ 0.354043] acpiphp: Slot [23] registered [ 0.354904] acpiphp: Slot [24] registered [ 0.356049] acpiphp: Slot [25] registered [ 0.356827] acpiphp: Slot [26] registered [ 0.358047] acpiphp: Slot [27] registered [ 0.358846] acpiphp: Slot [28] registered [ 0.359046] acpiphp: Slot [29] registered [ 0.359873] acpiphp: Slot [30] registered [ 0.361085] acpiphp: Slot [31] registered [ 0.361903] PCI host bridge to bus 0000:00 [ 0.363011] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.364010] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.365008] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.367008] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.369013] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.370009] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.371107] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.373540] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.376202] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.384589] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.388514] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.390008] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.391008] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.393007] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.394443] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.395472] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.397019] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.399565] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.402007] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.410554] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.414007] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.417200] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.421009] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.425010] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.435011] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.441033] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.445009] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.448007] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.458012] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.463409] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.468008] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.472007] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.483009] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.489043] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.492009] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.497007] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.506012] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.512124] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.516007] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.520008] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.533014] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.538701] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.542019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.545984] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.556015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.561762] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.563202] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.565207] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.566189] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.568110] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.570194] iommu: Default domain type: Passthrough [ 0.571504] SCSI subsystem initialized [ 0.572099] ACPI: bus type USB registered [ 0.572928] usbcore: registered new interface driver usbfs [ 0.574028] usbcore: registered new interface driver hub [ 0.575029] usbcore: registered new device driver usb [ 0.576064] pps_core: LinuxPPS API ver. 1 registered [ 0.576956] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.579016] PTP clock support registered [ 0.580064] EDAC MC: Ver: 3.0.0 [ 0.581213] PCI: Using ACPI for IRQ routing [ 0.582659] NetLabel: Initializing [ 0.584009] NetLabel: domain hash size = 128 [ 0.584875] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.586042] NetLabel: unlabeled traffic allowed by default [ 0.587081] vgaarb: loaded [ 0.588135] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.589006] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.595223] clocksource: Switched to clocksource kvm-clock [ 0.678033] VFS: Disk quotas dquot_6.6.0 [ 0.679533] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.680909] *** VALIDATE ramfs *** [ 0.681794] *** VALIDATE hugetlbfs *** [ 0.683326] pnp: PnP ACPI init [ 0.684914] pnp: PnP ACPI: found 6 devices [ 0.697667] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.699443] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.700523] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.701583] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.702711] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.703869] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.705356] NET: Registered protocol family 2 [ 0.706842] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.709752] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.711809] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.715507] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.717501] TCP: Hash tables configured (established 65536 bind 65536) [ 0.719141] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.720843] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.722560] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.724536] NET: Registered protocol family 1 [ 0.726025] RPC: Registered named UNIX socket transport module. [ 0.727186] RPC: Registered udp transport module. [ 0.727958] RPC: Registered tcp transport module. [ 0.728740] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.730061] NET: Registered protocol family 44 [ 0.730906] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.732191] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.733303] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.734605] PCI: CLS 0 bytes, default 64 [ 0.735635] Unpacking initramfs... [ 1.915302] debug: unmapping init [mem 0xffff9a2d7cc54000-0xffff9a2d7ffbffff] [ 1.920256] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 1.921549] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 1.924290] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.372140] Initialise system trusted keyrings [ 2.373607] Key type blacklist registered [ 2.375094] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.383705] zbud: loaded [ 2.386444] *** VALIDATE nfs *** [ 2.387489] *** VALIDATE nfs4 *** [ 2.388720] pstore: using deflate compression [ 2.391174] Platform Keyring initialized [ 2.467304] NET: Registered protocol family 38 [ 2.468726] Key type asymmetric registered [ 2.470127] Asymmetric key parser 'x509' registered [ 2.471665] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.474235] io scheduler mq-deadline registered [ 2.475715] io scheduler kyber registered [ 2.477211] io scheduler bfq registered [ 2.478691] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.481203] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.483844] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.486902] ACPI: Power Button [PWRF] [ 2.491679] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.499153] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.510309] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.516740] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.532685] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.559834] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.587799] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.592159] Non-volatile memory driver v1.3 [ 2.593237] Linux agpgart interface v0.103 [ 2.616527] virtio_blk virtio1: [vda] 134040 512-byte logical blocks (68.6 MB/65.4 MiB) [ 2.618290] vda: detected capacity change from 0 to 68628480 [ 2.628735] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.631554] vdb: detected capacity change from 0 to 1073741824 [ 2.645769] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.649088] vdc: detected capacity change from 0 to 2621440000 [ 2.659682] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.661513] vdd: detected capacity change from 0 to 2621440000 [ 2.673440] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.676420] vde: detected capacity change from 0 to 4294967296 [ 2.690103] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.693647] vdf: detected capacity change from 0 to 4294967296 [ 2.700225] libphy: Fixed MDIO Bus: probed [ 2.704924] usbcore: registered new interface driver usbserial_generic [ 2.707448] usbserial: USB Serial support registered for generic [ 2.709632] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.713456] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.715253] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.717557] mousedev: PS/2 mouse device common for all mice [ 2.720531] rtc_cmos 00:05: RTC can wake from S4 [ 2.722391] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.724134] rtc_cmos 00:05: registered as rtc0 [ 2.728607] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.728646] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.731367] intel_pstate: CPU model not supported [ 2.735565] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.740498] hid: raw HID events driver (C) Jiri Kosina [ 2.742556] usbcore: registered new interface driver usbhid [ 2.744527] usbhid: USB HID core driver [ 2.746069] drop_monitor: Initializing network drop monitor service [ 2.748392] Initializing XFRM netlink socket [ 2.750337] NET: Registered protocol family 10 [ 2.752928] Segment Routing with IPv6 [ 2.754374] NET: Registered protocol family 17 [ 2.756213] mpls_gso: MPLS GSO support [ 2.760782] RAS: Correctable Errors collector initialized. [ 2.762095] AVX version of gcm_enc/dec engaged. [ 2.763023] AES CTR mode by8 optimization enabled [ 2.819467] sched_clock: Marking stable (2819429653, 0)->(3408359651, -588929998) [ 2.822976] registered taskstats version 1 [ 2.824809] Loading compiled-in X.509 certificates [ 2.826598] zswap: loaded using pool lzo/zbud [ 2.869532] Key type big_key registered [ 2.880761] Key type encrypted registered [ 2.882301] ima: No TPM chip found, activating TPM-bypass! [ 2.884201] ima: Allocated hash algorithm: sha1 [ 2.885635] ima: No architecture policies found [ 2.887266] evm: Initialising EVM extended attributes: [ 2.888964] evm: security.selinux [ 2.890126] evm: security.ima [ 2.891157] evm: security.capability [ 2.892331] evm: HMAC attrs: 0x1 [ 2.894504] rtc_cmos 00:05: setting system clock to 2026-01-02 20:01:19 UTC (1767384079) [ 2.900849] debug: unmapping init [mem 0xffffffff97403000-0xffffffff975fffff] [ 2.904375] debug: unmapping init [mem 0xffffffff96182000-0xffffffff96458fff] [ 2.913096] Write protecting the kernel read-only data: 28672k [ 2.916344] debug: unmapping init [mem 0xffffffff94803000-0xffffffff949fffff] [ 2.919016] debug: unmapping init [mem 0xffffffff95114000-0xffffffff951fffff] [ 2.957936] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 2.967094] systemd[1]: Detected virtualization kvm. [ 2.969188] systemd[1]: Detected architecture x86-64. [ 2.971354] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 2.996641] systemd[1]: No hostname configured. [ 2.998520] systemd[1]: Set hostname to . [ 3.000948] random: systemd: uninitialized urandom read (16 bytes read) [ 3.003885] systemd[1]: Initializing machine ID from random generator. [ 3.063270] random: ln: uninitialized urandom read (6 bytes read) [ 3.147568] random: systemd: uninitialized urandom read (16 bytes read) [ 3.150306] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.158433] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ 3.164760] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... Starting Setup Virtual Console... Starting Journal Service... [ OK ] Reached target Paths. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.670994] device-mapper: uevent: version 1.0.3 [ 3.672920] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.257481] random: fast init done [ 4.294671] virtio_net virtio0 ens2: renamed from eth0 Starting dracut initqueue hook... [ 4.342969] scsi host0: ata_piix [ 4.358188] scsi host1: ata_piix [ 4.359990] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.362291] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.869940] dracut-initqueue[585]: RTNETLINK answers: File exists [ 9.402980] random: crng init done [ 9.404197] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 9.899245] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.117941] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.347808] SELinux: Disabled at runtime. [ 11.412856] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.420386] systemd[1]: Detected virtualization kvm. [ 11.421921] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.915220] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.918904] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.924089] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.929196] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.932348] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.937655] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.941187] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. Mounting POSIX Message Queue File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. [ 12.031574] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd File Systems. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Mounting Kernel Debug File System... [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.354557] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 12.696252] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.702910] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.860286] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.888822] EDAC sbridge: Ver: 1.1.2 [ 14.504503] Key type dns_resolver registered [ 14.820741] NFS: Registering the id_resolver key type [ 14.824253] Key type id_resolver registered [ 14.826274] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg406-server login: [ 36.070126] libcfs: loading out-of-tree module taints kernel. [ 36.085826] Key type ._llcrypt registered [ 36.086807] Key type .llcrypt registered [ 36.120228] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 43.158968] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 43.664543] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 43.669742] alg: No test for adler32 (adler32-zlib) [ 44.630218] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 44.907437] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 46.512169] Key type lgssc registered [ 47.092299] Lustre: Echo OBD driver; http://www.lustre.org/ [ 52.947351] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 53.524960] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 56.568475] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 59.050308] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 61.543594] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 66.689611] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 70.885532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 70.907971] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 70.913781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 71.999755] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 72.010332] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 72.046653] Lustre: lustre-MDT0000: new disk, initializing [ 72.071847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 72.080068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 73.318162] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 78.552353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 78.581704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 78.611930] Lustre: 6481:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 78.624821] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 78.626457] Lustre: Skipped 1 previous similar message [ 78.676517] Lustre: lustre-MDT0001: new disk, initializing [ 78.699076] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 78.709264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 78.711361] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 79.992068] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 82.351659] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 85.761202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 85.783560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 85.898550] Lustre: lustre-OST0000: new disk, initializing [ 85.901345] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 85.921871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 87.859886] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 88.879501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 88.881782] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 88.915960] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 93.745810] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 93.775334] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 93.823069] Lustre: lustre-OST0001: new disk, initializing [ 93.826279] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 93.848778] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 95.887495] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 97.204880] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 97.212583] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 97.226369] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 102.113166] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 108.761276] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 115.664263] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing check_logdir /tmp/testlogs/ [ 117.367548] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing yml_node [ 118.779910] Lustre: DEBUG MARKER: Client: 2.17.0.RC4 [ 119.806790] Lustre: DEBUG MARKER: MDS: 2.17.0.RC4 [ 120.824521] Lustre: DEBUG MARKER: OSS: 2.17.0.RC4 [ 121.477228] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Jan 2 15:03:17 EST 2026 [ 128.863549] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 129.562819] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 143.328890] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 143.329448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 143.331958] Lustre: Skipped 3 previous similar messages [ 143.333410] Lustre: Skipped 2 previous similar messages [ 147.712062] LustreError: 12185:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 147.741874] Lustre: server umount lustre-MDT0000 complete [ 149.441113] LustreError: 10249:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767384226 with bad export cookie 8271898750564268064 [ 149.443793] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 149.448535] LustreError: 10249:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 153.568986] LustreError: 10270:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 153.569575] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 153.570263] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 153.570275] Lustre: Skipped 1 previous similar message [ 153.585708] LustreError: 10270:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 153.595088] Lustre: Skipped 1 previous similar message [ 154.594262] LustreError: 6488:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 154.594310] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 154.600256] LustreError: 6488:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 154.602601] Lustre: Skipped 2 previous similar messages [ 155.581972] LustreError: 12387:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 155.583710] LustreError: 12387:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 155.619729] Lustre: server umount lustre-MDT0001 complete [ 156.949127] LustreError: 12588:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 156.954217] LustreError: 12588:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 156.981674] Lustre: server umount lustre-OST0000 complete [ 158.286534] Lustre: server umount lustre-OST0001 complete [ 160.343450] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 163.163467] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 166.500067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 168.820140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 171.145309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 173.434809] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 176.246672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 176.277123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 176.383138] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 176.396887] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 176.441979] Lustre: lustre-MDT0000: new disk, initializing [ 176.478369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 176.487241] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 178.042107] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 182.169835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 182.191033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 182.224928] Lustre: 14968:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 182.340911] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 183.679814] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 186.758077] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 188.035340] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 190.397422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 190.421687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 190.511488] Lustre: lustre-OST0000: new disk, initializing [ 190.512682] Lustre: Skipped 1 previous similar message [ 190.514230] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 190.515634] Lustre: Skipped 2 previous similar messages [ 190.541267] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 190.542917] Lustre: Skipped 1 previous similar message [ 191.993747] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 191.996164] Lustre: Skipped 1 previous similar message [ 191.997767] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 192.035292] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 192.410753] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 195.571365] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 196.892488] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 196.987791] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 198.379317] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 198.453155] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 202.210241] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 202.210690] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 202.212091] LustreError: Skipped 1 previous similar message [ 202.215965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 202.218445] Lustre: Skipped 1 previous similar message [ 205.167102] LustreError: 17906:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 205.169053] LustreError: 17906:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 205.202430] Lustre: server umount lustre-OST0000 complete [ 207.329870] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 207.330944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 207.333349] Lustre: Skipped 2 previous similar messages [ 207.338718] Lustre: Skipped 1 previous similar message [ 212.741141] LustreError: 18107:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 212.744148] LustreError: 18107:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 212.771901] Lustre: server umount lustre-MDT0000 complete [ 214.070628] LustreError: 15748:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767384290 with bad export cookie 8271898750564269583 [ 214.074699] LustreError: 15748:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 214.076893] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 214.195572] Lustre: server umount lustre-MDT0001 complete [ 216.387351] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 15:04:52 (1767384292) [ 222.945129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 223.093620] LustreError: 19441:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 223.116916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 224.444911] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 227.414995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 228.856151] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 230.574834] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 231.713230] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 232.483833] Lustre: Failing over lustre-MDT0000 [ 232.487926] LustreError: 19470:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 232.491012] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 232.491277] LustreError: 19470:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 232.491813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 232.491820] Lustre: Skipped 2 previous similar messages [ 232.497025] Lustre: Skipped 1 previous similar message [ 232.499380] LustreError: 19470:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -5 [ 232.571069] LustreError: 20785:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 232.572845] LustreError: 20785:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 232.595063] Lustre: server umount lustre-MDT0000 complete [ 237.537053] LustreError: 20153:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 237.544333] LustreError: 20153:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 243.680981] LustreError: 20153:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 243.685275] LustreError: 20153:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 245.534276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 245.568131] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 246.930759] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 250.019881] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 250.850639] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 250.853567] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 250.862374] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 254.050335] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 254.058166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 254.061564] Lustre: Skipped 2 previous similar messages [ 258.107300] LustreError: 21765:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 258.109189] LustreError: 21765:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 258.131613] Lustre: server umount lustre-MDT0000 complete [ 259.353231] LustreError: 19421:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767384335 with bad export cookie 8271898750564270990 [ 259.355958] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 259.356210] LustreError: 19421:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 259.472899] Lustre: server umount lustre-MDT0001 complete [ 262.335974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 262.495228] LustreError: 22441:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 262.522132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 262.523751] Lustre: Skipped 2 previous similar messages [ 263.853198] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 266.721347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 268.137814] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 269.896208] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 271.092800] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 271.776212] Lustre: Failing over lustre-MDT0000 [ 271.779492] LustreError: 22470:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 271.781633] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 271.784891] LustreError: 22470:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 271.788671] Lustre: Skipped 1 previous similar message [ 271.795698] LustreError: 22470:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 271.887189] Lustre: server umount lustre-MDT0000 complete [ 282.081023] LustreError: 23152:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 282.086165] LustreError: 23152:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 284.777272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 284.813346] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 286.151171] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 289.268395] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 290.273683] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 290.276585] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 290.279148] Lustre: Skipped 1 previous similar message [ 290.284518] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 291.245889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 291.248013] Lustre: Skipped 3 previous similar messages [ 297.347929] LustreError: 24762:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 297.349878] LustreError: 24762:0:(obd_class.h:479:obd_check_dev()) Skipped 19 previous similar messages [ 297.374367] Lustre: server umount lustre-MDT0000 complete [ 298.506973] LustreError: 22422:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767384375 with bad export cookie 8271898750564271921 [ 298.509789] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 298.510094] LustreError: 22422:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 298.617031] Lustre: server umount lustre-MDT0001 complete [ 301.539875] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 15:06:18 (1767384378) [ 306.931889] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 310.105725] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 313.477045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 315.673750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 317.731726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 319.798523] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 323.081217] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 327.035404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 327.060642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 327.153612] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 327.165259] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 327.212898] Lustre: lustre-MDT0000: new disk, initializing [ 327.241537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 327.243418] Lustre: Skipped 2 previous similar messages [ 327.248841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 328.647034] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 332.758830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 332.787414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 332.818741] Lustre: 28423:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 332.920570] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 334.210701] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 337.469954] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 338.617127] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 341.304495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 341.340195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 342.670586] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 342.698936] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 342.705527] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 343.404560] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 346.546933] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 348.989504] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 353.255116] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 353.256921] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 353.262327] Lustre: Skipped 4 previous similar messages [ 353.270073] LustreError: Skipped 1 previous similar message [ 355.309094] Lustre: server umount lustre-OST0000 complete [ 358.372247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 358.376134] Lustre: Skipped 5 previous similar messages [ 363.327848] LustreError: 31265:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 363.329570] LustreError: 31265:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 363.358644] Lustre: server umount lustre-MDT0000 complete [ 363.489218] LustreError: 28435:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 363.494130] LustreError: 28435:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 364.869886] LustreError: 30175:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767384441 with bad export cookie 8271898750564272327 [ 364.876161] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 364.879779] LustreError: 30175:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 365.113186] Lustre: server umount lustre-MDT0001 complete [ 369.473284] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 370.655815] Key type lgssc unregistered [ 370.788603] LNet: 31998:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 370.791782] LNetError: 31998:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 370.802492] LNet: Removed LNI 192.168.204.106@tcp [ 371.242120] Key type .llcrypt unregistered [ 371.243836] Key type ._llcrypt unregistered [ 379.992582] Key type ._llcrypt registered [ 379.993770] Key type .llcrypt registered [ 380.024837] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 380.326859] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 380.366696] alg: No test for adler32 (adler32-zlib) [ 381.230519] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 381.313276] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 382.888221] Key type lgssc registered [ 383.306755] Lustre: Echo OBD driver; http://www.lustre.org/ [ 385.977447] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 385.988416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 387.191208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 388.574532] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 392.131382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 392.249724] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 393.512514] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 395.390215] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 396.870529] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 399.370824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 399.498495] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 401.491889] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 401.581709] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 403.204584] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 419.301109] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 15:08:15 (1767384495) [ 420.039879] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 420.717676] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 423.460323] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 15:08:19 (1767384499) [ 427.506673] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 15:08:23 (1767384503) [ 427.956465] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 431.961178] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 431.965538] Lustre: Skipped 2 previous similar messages [ 432.873578] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 15:08:29 (1767384509) [ 436.498307] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 436.503106] Lustre: Skipped 4 previous similar messages [ 443.591824] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 15:08:40 (1767384520) [ 444.537349] Lustre: *** cfs_fail_loc=131b, val=2*** [ 445.705528] Lustre: *** cfs_fail_loc=131b, val=2*** [ 449.004791] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 15:08:45 (1767384525) [ 449.840642] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 449.844116] Lustre: Skipped 9 previous similar messages [ 454.792948] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 15:08:51 (1767384531) [ 459.237269] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 459.238759] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 459.242816] Lustre: Skipped 1 previous similar message [ 459.243176] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 459.245995] Lustre: Skipped 1 previous similar message [ 464.354566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 464.357889] Lustre: Skipped 1 previous similar message [ 465.075150] LustreError: 39939:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 465.127734] Lustre: server umount lustre-OST0000 complete [ 468.961445] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 468.967193] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 468.975520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 474.594181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 474.598299] Lustre: Skipped 3 previous similar messages [ 479.714162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 479.718017] Lustre: Skipped 1 previous similar message [ 481.248206] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 481.306512] LustreError: 40142:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 481.310370] LustreError: 40142:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 481.343767] Lustre: server umount lustre-MDT0000 complete [ 483.122191] LustreError: 36613:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767384559 with bad export cookie 14855968100803881798 [ 483.128042] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 483.130549] LustreError: 36613:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 483.226883] LustreError: 40345:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 483.230283] LustreError: 40345:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 483.276267] Lustre: server umount lustre-MDT0001 complete [ 487.501897] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 488.651153] Key type lgssc unregistered [ 488.767280] LNet: 40876:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 488.769293] LNetError: 40876:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 488.776301] LNet: Removed LNI 192.168.204.106@tcp [ 489.046545] Key type .llcrypt unregistered [ 489.047520] Key type ._llcrypt unregistered [ 495.993043] Key type ._llcrypt registered [ 495.994092] Key type .llcrypt registered [ 496.034940] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 502.822393] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 503.163823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 503.178176] alg: No test for adler32 (adler32-zlib) [ 504.024098] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 504.114722] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 505.696192] Key type lgssc registered [ 506.244196] Lustre: Echo OBD driver; http://www.lustre.org/ [ 509.455283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 512.244126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 515.372758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 518.608888] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 521.931071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 521.964272] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 521.973664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 523.068297] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 523.083980] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 523.118179] Lustre: lustre-MDT0000: new disk, initializing [ 523.148374] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 523.155519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 524.927172] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 530.211404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 530.250147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 530.308419] Lustre: 43978:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 530.349489] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 530.353226] Lustre: Skipped 1 previous similar message [ 530.407128] Lustre: lustre-MDT0001: new disk, initializing [ 530.454615] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 530.474799] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 530.481497] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 532.159731] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 536.246492] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 538.022673] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 541.248354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 541.290036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 541.434352] Lustre: lustre-OST0000: new disk, initializing [ 541.438106] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 541.487422] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 544.005544] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 547.330212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 547.335110] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 547.377091] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 548.094164] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 550.089328] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 550.198189] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 551.908641] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 552.012885] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 557.541636] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 557.543694] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 557.546262] LustreError: Skipped 1 previous similar message [ 557.555245] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 557.560410] Lustre: Skipped 1 previous similar message [ 562.662211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 562.665939] Lustre: Skipped 1 previous similar message [ 567.264149] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 567.289140] LustreError: 46918:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 567.347117] Lustre: server umount lustre-OST0000 complete [ 571.361549] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 571.366707] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 571.374650] Lustre: Skipped 1 previous similar message [ 571.378182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 572.899002] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 575.248568] LustreError: 47120:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 575.252034] LustreError: 47120:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 575.283293] Lustre: server umount lustre-MDT0000 complete [ 576.768129] LustreError: 43969:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767384653 with bad export cookie 14812926617961547287 [ 576.770971] LustreError: 43969:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 576.771540] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 576.835123] LustreError: 47322:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 576.836822] LustreError: 47322:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 576.882493] Lustre: server umount lustre-MDT0001 complete [ 580.861421] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 15:10:57 (1767384657) [ 583.625291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 583.839721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 585.171105] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 588.976556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 589.140738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 590.697155] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 593.224410] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 594.914560] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 597.233299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 597.342663] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 599.174963] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 600.897233] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 613.051525] LustreError: 50255:0:(llog.c:441:llog_init_handle()) llog type is not specified! [ 613.342243] Lustre: Modifying parameter general.timeout=1 in log params [ 614.486677] Lustre: Modifying parameter general.timeout=5 in log params [ 614.488042] Lustre: Skipped 3 previous similar messages [ 616.549098] Lustre: Modifying parameter general.timeout=12 in log params [ 616.550558] Lustre: Skipped 6 previous similar messages [ 620.585971] Lustre: Modifying parameter general.timeout=27 in log params [ 620.587616] Lustre: Skipped 14 previous similar messages [ 628.861673] Lustre: Modifying parameter general.timeout=53 in log params [ 628.863435] Lustre: Skipped 25 previous similar messages [ 645.024614] Lustre: Modifying parameter general.timeout=109 in log params [ 645.026326] Lustre: Skipped 55 previous similar messages [ 655.565268] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 15:12:12 (1767384732) [ 656.817435] Lustre: 57153:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 658.120641] Lustre: 57349:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 658.124417] Lustre: 57349:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 1 previous similar message [ 660.177804] Lustre: 57643:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 660.181127] Lustre: 57643:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 2 previous similar messages [ 669.263117] Lustre: 59000:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 669.270226] LustreError: 59000:0:(mgs_handler.c:1129:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 670.512650] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 15:12:27 (1767384747) [ 676.323644] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 676.324732] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 676.325558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 676.325566] Lustre: Skipped 3 previous similar messages [ 676.328856] Lustre: Skipped 2 previous similar messages [ 680.770672] LustreError: 59697:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 680.772545] LustreError: 59697:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 680.800778] Lustre: server umount lustre-MDT0000 complete [ 681.445505] LustreError: 50016:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 681.449679] LustreError: 50016:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 682.089747] LustreError: 50306:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767384758 with bad export cookie 14812926617961548155 [ 682.096003] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 682.233125] Lustre: server umount lustre-MDT0001 complete [ 693.880501] LustreError: 60098:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 693.884323] LustreError: 60098:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 693.952902] Lustre: server umount lustre-OST0000 complete [ 696.963681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 698.571766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 700.179796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 701.496766] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 703.735592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 703.827102] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 703.836768] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 703.840161] Lustre: Skipped 43 previous similar messages [ 703.931498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 705.168938] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 708.665453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 708.699815] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 708.710450] Lustre: 61547:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 710.231071] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 712.401907] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 714.023138] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 716.510743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 716.585145] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 718.830812] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 720.801595] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 726.814470] Lustre: 63784:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 729.574692] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 729.577145] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 729.584942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 729.586520] Lustre: Skipped 2 previous similar messages [ 730.995203] LustreError: 64031:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 730.997497] LustreError: 64031:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 731.039415] Lustre: server umount lustre-OST0000 complete [ 734.581454] Lustre: server umount lustre-MDT0000 complete [ 734.690592] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 734.692468] LustreError: 63734:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 734.696418] LustreError: Skipped 1 previous similar message [ 734.706465] LustreError: 63734:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 736.321068] LustreError: 61539:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767384812 with bad export cookie 14812926617961550423 [ 736.326166] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 736.327676] LustreError: 61539:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 736.472558] Lustre: server umount lustre-MDT0001 complete [ 740.737905] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 741.721918] Key type lgssc unregistered [ 741.872652] LNet: 64971:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 741.877455] LNetError: 64971:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 741.891595] LNet: Removed LNI 192.168.204.106@tcp [ 742.226326] Key type .llcrypt unregistered [ 742.227555] Key type ._llcrypt unregistered [ 753.207046] Key type ._llcrypt registered [ 753.208067] Key type .llcrypt registered [ 753.251350] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 753.599985] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 753.633103] alg: No test for adler32 (adler32-zlib) [ 754.493170] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 754.594354] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 756.176229] Key type lgssc registered [ 756.620108] Lustre: Echo OBD driver; http://www.lustre.org/ [ 759.712267] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 759.723431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 760.921954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 762.294372] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 765.397676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 765.600286] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 766.939767] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 768.921164] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 770.134858] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 772.733564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 772.948738] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 775.108497] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 776.916143] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 790.299141] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 796.129540] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 796.132837] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 796.138486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 796.643780] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 796.646130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 796.651584] Lustre: Skipped 1 previous similar message [ 799.169438] LustreError: 69074:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 799.198692] Lustre: server umount lustre-MDT0000 complete [ 800.627176] LustreError: 66325:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767384877 with bad export cookie 1139916509621924758 [ 800.630606] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 800.710874] LustreError: 69277:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 800.712575] LustreError: 69277:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 800.749995] Lustre: server umount lustre-MDT0001 complete [ 812.282539] LustreError: 69477:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 812.287119] LustreError: 69477:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 812.351572] Lustre: server umount lustre-OST0000 complete [ 815.750477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 817.444803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 819.052972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 820.532368] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 822.900015] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 822.997276] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 823.009290] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 823.093921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 824.376536] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 827.830019] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 827.868077] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 827.878199] Lustre: 70927:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 828.072543] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 829.414373] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 831.292715] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 832.661300] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 835.085790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 835.194836] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 837.353708] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 839.223507] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 845.098780] Lustre: 73166:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 845.808683] Lustre: Modifying parameter general.jobid_var=disable in log params [ 851.428177] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 851.428784] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 851.430993] LustreError: Skipped 1 previous similar message [ 851.434467] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 851.436509] Lustre: Skipped 2 previous similar messages [ 853.426558] LustreError: 73414:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 853.430387] LustreError: 73414:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 853.485224] Lustre: server umount lustre-OST0000 complete [ 856.545560] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 856.546410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 856.548588] Lustre: Skipped 2 previous similar messages [ 856.551368] Lustre: Skipped 1 previous similar message [ 861.665571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 861.668195] Lustre: Skipped 1 previous similar message [ 869.344114] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 869.384111] LustreError: 73621:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 869.385821] LustreError: 73621:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 869.409718] Lustre: server umount lustre-MDT0000 complete [ 870.659179] LustreError: 70918:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767384947 with bad export cookie 1139916509621925570 [ 870.662058] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 870.664591] LustreError: 70918:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 870.786104] Lustre: server umount lustre-MDT0001 complete [ 874.263095] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 875.337534] Key type lgssc unregistered [ 875.474603] LNet: 74354:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 875.477346] LNetError: 74354:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 875.486419] LNet: Removed LNI 192.168.204.106@tcp [ 875.795931] Key type .llcrypt unregistered [ 875.797046] Key type ._llcrypt unregistered [ 884.731306] Key type ._llcrypt registered [ 884.732690] Key type .llcrypt registered [ 884.772710] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 885.046952] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 885.052603] alg: No test for adler32 (adler32-zlib) [ 885.899176] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 885.975482] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 887.552153] Key type lgssc registered [ 887.928943] Lustre: Echo OBD driver; http://www.lustre.org/ [ 890.823375] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 890.830447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 892.040785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 893.626277] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 897.106708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 897.330591] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 898.799165] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 900.725469] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 901.937358] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 904.482292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 904.743944] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 906.929527] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 909.009038] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 921.840725] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 922.666027] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 922.668776] Lustre: Skipped 1 previous similar message [ 923.881322] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 923.885073] Lustre: Skipped 2 previous similar messages [ 926.293238] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 926.296883] Lustre: Skipped 5 previous similar messages [ 930.439280] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 930.442079] Lustre: Skipped 11 previous similar messages [ 938.472173] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 938.474227] Lustre: Skipped 23 previous similar messages [ 954.576318] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 954.578177] Lustre: Skipped 42 previous similar messages [ 962.199353] Lustre: 83485:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 963.522422] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 15:17:19 (1767385039) [ 964.236187] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 965.105641] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 15:17:21 (1767385041) [ 968.164175] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 968.166051] LustreError: Skipped 1 previous similar message [ 968.166081] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 968.169170] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 968.170445] Lustre: Skipped 1 previous similar message [ 972.340238] LustreError: 83778:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 972.406123] Lustre: server umount lustre-OST0000 complete [ 974.305365] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 974.311059] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 974.319855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 974.322755] Lustre: Skipped 1 previous similar message [ 978.401857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 978.402966] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 980.425072] LustreError: 83981:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 980.428048] LustreError: 83981:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 980.481159] Lustre: server umount lustre-MDT0000 complete [ 982.201702] LustreError: 77355:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767385058 with bad export cookie 1485082352636429949 [ 982.209679] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 982.302163] LustreError: 84183:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 982.306131] LustreError: 84183:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 982.379566] Lustre: server umount lustre-MDT0001 complete [ 986.449807] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 987.632245] Key type lgssc unregistered [ 987.768334] LNet: 84714:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 987.770661] LNetError: 84714:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 987.779366] LNet: Removed LNI 192.168.204.106@tcp [ 988.170515] Key type .llcrypt unregistered [ 988.172725] Key type ._llcrypt unregistered [ 989.493889] Key type ._llcrypt registered [ 989.495245] Key type .llcrypt registered [ 989.531627] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 990.942772] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules [ 991.238936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 991.302357] alg: No test for adler32 (adler32-zlib) [ 992.142552] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 992.223156] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 993.808247] Key type lgssc registered [ 994.300419] Lustre: Echo OBD driver; http://www.lustre.org/ [ 994.665224] LustreError: 85832:0:(lu_object.c:1438:lu_context_key_register()) cfs_race id 60d sleeping [ 1003.192185] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 1006.854675] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1016.161594] LustreError: 85832:0:(lu_object.c:1438:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1016.172976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1017.460298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1019.127244] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1022.635307] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 15:18:18 (1767385098) [ 1024.409155] LustreError: 86870:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1024.412129] LustreError: 86870:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1024.414519] LustreError: 86870:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 1024.482799] LustreError: 87642:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1024.535913] Lustre: server umount lustre-MDT0000 complete [ 1029.401119] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 1030.438072] Key type lgssc unregistered [ 1030.603340] LNet: 88328:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1030.605399] LNetError: 88328:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1030.616290] LNet: Removed LNI 192.168.204.106@tcp [ 1030.938639] Key type .llcrypt unregistered [ 1030.940853] Key type ._llcrypt unregistered [ 1039.526512] Key type ._llcrypt registered [ 1039.528084] Key type .llcrypt registered [ 1039.573884] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 1039.959537] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1039.967871] alg: No test for adler32 (adler32-zlib) [ 1040.816386] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 1040.905811] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 1042.504102] Key type lgssc registered [ 1043.055372] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1046.616087] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1046.627058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1047.859900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1049.450302] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1053.179359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1053.366735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1054.909708] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1057.428677] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1059.163484] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1061.507376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1061.722936] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1064.198143] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1066.070618] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1102.225489] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 15:19:38 (1767385178) [ 1102.945490] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1103.816751] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 15:19:40 (1767385180) [ 1108.968402] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1108.971172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1108.974041] Lustre: Skipped 2 previous similar messages [ 1108.977074] Lustre: Skipped 1 previous similar message [ 1111.311980] LustreError: 92431:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1111.344159] Lustre: server umount lustre-MDT0000 complete [ 1112.586678] LustreError: 89483:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767385189 with bad export cookie 187749357462035558 [ 1112.592706] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1112.593613] LustreError: 89483:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1112.695389] LustreError: 92634:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1112.699109] LustreError: 92634:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1112.775268] Lustre: server umount lustre-MDT0001 complete [ 1124.853042] LustreError: 92835:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1124.855465] LustreError: 92835:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1124.973471] Lustre: server umount lustre-OST0000 complete [ 1129.566210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1129.772082] LustreError: 93464:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1129.779262] LustreError: 93464:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1129.850704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1131.537154] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1135.075143] LustreError: 93464:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1135.121796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1135.404048] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1137.192773] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1139.256447] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1140.795481] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1143.791162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1146.830699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1146.860894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1146.948437] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1146.952955] LustreError: 93453:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1146.969382] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1146.972556] LustreError: 95324:0:(tgt_mount.c:2572:server_fill_super()) Unable to start targets: -98 [ 1146.975233] LustreError: 95324:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 1146.977253] LustreError: 95324:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 1147.004139] Lustre: server umount lustre-OST0000 complete [ 1147.005414] LustreError: 95324:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1151.002882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1151.136087] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1151.140986] LustreError: 93453:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1151.147521] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1151.152488] LustreError: 95788:0:(tgt_mount.c:2572:server_fill_super()) Unable to start targets: -98 [ 1151.156827] LustreError: 95788:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 1151.160278] LustreError: 95788:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 1151.188075] Lustre: server umount lustre-OST0000 complete [ 1151.189799] LustreError: 95788:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1152.413956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1154.884635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1154.979858] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1154.983057] Lustre: Found index 0 for lustre-OST0000, updating log [ 1154.987002] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1155.018360] Lustre: lustre-OST0000: new disk, initializing [ 1155.021775] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1155.142211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1157.928629] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1159.710101] Lustre: Failing over lustre-OST0000 [ 1159.737177] LustreError: 96869:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1159.740990] LustreError: 96869:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1159.795552] Lustre: server umount lustre-OST0000 complete [ 1161.672563] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1161.680954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1161.684743] Lustre: Skipped 1 previous similar message [ 1166.305408] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1166.309985] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1166.318081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1167.827831] LustreError: 97070:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1167.831860] LustreError: 97070:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1167.870723] Lustre: server umount lustre-MDT0000 complete [ 1169.657194] LustreError: 93445:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767385246 with bad export cookie 187749357462038288 [ 1169.663401] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1169.664689] LustreError: 93445:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1169.818974] Lustre: server umount lustre-MDT0001 complete [ 1173.876293] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 15:20:50 (1767385250) [ 1180.995887] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 1185.046473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1185.249572] LustreError: 98638:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1185.296605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1186.979413] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1190.369984] LustreError: 98638:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1190.385430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1192.309896] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1193.436485] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 1196.458915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1199.057173] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1202.942983] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1202.975071] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1203.014596] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 1203.067046] Lustre: lustre-OST0001: new disk, initializing [ 1203.069824] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1205.548263] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1207.130710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1207.137096] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1207.189931] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1214.737428] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1221.157894] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1230.824293] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1230.826715] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 1230.827839] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1230.827847] Lustre: Skipped 1 previous similar message [ 1230.832604] Lustre: Skipped 1 previous similar message [ 1234.193738] LustreError: 101879:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 1234.197616] LustreError: 101879:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 1234.230190] Lustre: server umount lustre-MDT0001 complete [ 1235.566150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1235.942206] LustreError: 98637:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1235.951442] LustreError: 98637:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1238.766414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1238.822488] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1238.825306] Lustre: Found index 1 for lustre-MDT0001, updating log [ 1238.848310] Lustre: 98627:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1239.175572] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1239.177734] Lustre: Skipped 3 previous similar messages [ 1240.808083] LustreError: 102648:0:(obd_mount.c:188:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 1240.812048] LustreError: 102648:0:(tgt_mount.c:669:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 1240.817167] LustreError: 98629:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 1240.833774] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1240.855401] LustreError: 102661:0:(obd_config.c:2001:class_config_llog_handler()) MGC192.168.204.106@tcp: cfg command failed: rc = -17 [ 1240.858899] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1240.858899] [ 1244.651762] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1245.207557] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 15:22:01 (1767385321) [ 1252.495568] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 1260.286258] Lustre: 104485:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1269.976987] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1271.715864] Lustre: 105617:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1331.168736] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1331.171504] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1331.175225] Lustre: Skipped 1 previous similar message [ 1331.176968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1331.178969] Lustre: Skipped 3 previous similar messages [ 1335.775318] LustreError: 106071:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 1335.777593] LustreError: 106071:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1335.807459] Lustre: server umount lustre-MDT0000 complete [ 1336.801332] LustreError: 101372:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1336.810454] LustreError: 101372:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1337.378889] LustreError: 100913:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767385413 with bad export cookie 187749357462039128 [ 1337.380760] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1337.383192] LustreError: 100913:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1337.541168] Lustre: server umount lustre-MDT0001 complete [ 1349.141546] Lustre: server umount lustre-OST0000 complete [ 1361.273248] Lustre: server umount lustre-OST0001 complete [ 1363.748885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1367.830321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1368.273838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1371.882465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1376.027992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1376.431591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1384.511521] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 1388.922794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1388.933694] Lustre: lustre-MDT0000: reset Object Index mappings [ 1389.182548] LustreError: 109549:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1389.229781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1390.995083] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1394.371065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1394.381876] Lustre: lustre-MDT0001: reset Object Index mappings [ 1396.243395] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1397.592605] Lustre: 110663:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1400.104544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1402.731947] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1404.525564] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:195 to 0x280000400:225) [ 1406.833400] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1408.638221] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 1409.383941] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1413.749142] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 1413.754323] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:354 to 0x280000401:386) [ 1418.748255] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1420.657863] Lustre: 112519:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1446.641577] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 15:25:23 (1767385523) [ 1475.557850] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1475.562858] Lustre: Skipped 5 previous similar messages [ 1475.563029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1475.567220] Lustre: Skipped 6 previous similar messages [ 1481.639583] LustreError: 112868:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1481.642253] LustreError: 112868:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 1481.689071] Lustre: server umount lustre-MDT0000 complete [ 1483.508857] LustreError: 109528:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767385560 with bad export cookie 187749357462085923 [ 1483.515915] LustreError: 109528:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1483.521970] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1485.793692] LustreError: 109543:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1485.802706] LustreError: 109543:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1489.899513] Lustre: server umount lustre-MDT0001 complete [ 1497.815363] Lustre: server umount lustre-OST0000 complete [ 1505.788253] Lustre: server umount lustre-OST0001 complete [ 1508.691721] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 1513.383831] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 1518.260440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1521.541747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1524.535879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1527.686796] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1530.980128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1531.015182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1531.127530] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1531.146525] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1531.202866] Lustre: lustre-MDT0000: new disk, initializing [ 1531.238303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1531.241743] Lustre: Skipped 3 previous similar messages [ 1531.251161] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1532.975015] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1536.792640] Lustre: server umount lustre-MDT0000 complete [ 1538.113924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1541.411458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1541.435224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1541.537738] Lustre: Found index 0 for lustre-MDT0000, updating log [ 1541.542289] Lustre: 116741:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1541.546762] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 1543.219106] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1545.604645] Lustre: server umount lustre-MDT0000 complete [ 1549.639988] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 15:27:05 (1767385625) [ 1550.423866] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 1551.285486] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 15:27:07 (1767385627) [ 1557.444564] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 1561.496716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1563.230441] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1566.345255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1566.373925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1566.421202] Lustre: 118759:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1566.456278] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1566.459463] Lustre: Skipped 1 previous similar message [ 1566.506489] Lustre: lustre-MDT0001: new disk, initializing [ 1566.562155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1566.566660] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1567.926775] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1570.710979] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 1573.965703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1574.003501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1574.137583] Lustre: lustre-OST0000: new disk, initializing [ 1574.141687] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1575.950984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1575.958790] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1576.015908] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1576.632695] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1581.746557] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1581.776585] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1583.524853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1583.533561] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1583.574235] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1584.513487] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1589.560608] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1590.989743] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1765.527814] Lustre: 127543:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 1765.533815] Lustre: 127543:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 1765.538480] Lustre: 127543:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 1765.543296] Lustre: 127543:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 1765.548277] Lustre: 127543:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 1765.552889] Lustre: 127543:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1878.975953] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 15:32:35 (1767385955) [ 1890.788191] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1890.791870] Lustre: Skipped 4 previous similar messages [ 1890.793536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1890.797053] Lustre: Skipped 11 previous similar messages [ 1896.993915] LustreError: 127756:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 1896.997752] LustreError: 127756:0:(obd_class.h:479:obd_check_dev()) Skipped 33 previous similar messages [ 1897.094818] Lustre: server umount lustre-MDT0000 complete [ 1898.915699] LustreError: 119484:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767385975 with bad export cookie 187749357462142658 [ 1898.920202] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1898.922651] LustreError: 119484:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1901.027095] LustreError: 121973:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1901.036859] LustreError: 121973:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1913.312088] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1913.430765] Lustre: server umount lustre-MDT0001 complete [ 1921.102777] Lustre: server umount lustre-OST0000 complete [ 1928.689688] Lustre: server umount lustre-OST0001 complete [ 1931.549389] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 1935.689582] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 1940.023667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1942.956397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1945.287907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1948.292104] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1951.177864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1951.200594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1951.282262] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1951.292114] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1951.293545] Lustre: Skipped 1 previous similar message [ 1951.327130] Lustre: lustre-MDT0000: new disk, initializing [ 1951.328284] Lustre: Skipped 1 previous similar message [ 1951.351530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1951.353189] Lustre: Skipped 5 previous similar messages [ 1951.358356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1952.853702] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1957.771370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1957.795126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1957.824550] Lustre: 130540:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1957.919721] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1959.178696] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1963.127438] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1964.802234] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1968.160034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1968.212408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1968.349609] Lustre: lustre-OST0000: new disk, initializing [ 1968.351988] Lustre: Skipped 1 previous similar message [ 1968.354848] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1968.357785] Lustre: Skipped 2 previous similar messages [ 1970.070505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1970.076159] Lustre: Skipped 1 previous similar message [ 1970.080880] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1970.145553] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1971.156462] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 1974.763520] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1978.339527] Lustre: lustre-MDD0000: changelog on [ 1979.412743] Lustre: lustre-MDD0001: changelog on [ 2545.932584] Lustre: lustre-MDD0001: changelog off [ 2546.807327] Lustre: 133631:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 2547.247940] Lustre: lustre-MDD0000: changelog off [ 2553.828139] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2553.829408] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2553.832912] LustreError: Skipped 1 previous similar message [ 2553.840282] Lustre: Skipped 3 previous similar messages [ 2553.841242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2553.849280] Lustre: Skipped 13 previous similar messages [ 2555.507135] LustreError: 139920:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2555.511101] LustreError: 139920:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 2555.579459] Lustre: server umount lustre-OST0000 complete [ 2558.947159] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2558.948622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2558.954712] Lustre: Skipped 2 previous similar messages [ 2558.961015] Lustre: Skipped 1 previous similar message [ 2563.546932] Lustre: server umount lustre-MDT0000 complete [ 2564.066414] LustreError: 130551:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2564.075291] LustreError: 130551:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2565.249727] LustreError: 131323:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767386641 with bad export cookie 187749357462947959 [ 2565.256579] LustreError: 131323:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2565.256718] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2565.397136] Lustre: server umount lustre-MDT0001 complete [ 2569.866804] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 2571.339137] Key type lgssc unregistered [ 2571.502712] LNet: 140853:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2571.507432] LNetError: 140853:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2571.516620] LNet: Removed LNI 192.168.204.106@tcp [ 2571.982174] Key type .llcrypt unregistered [ 2571.984489] Key type ._llcrypt unregistered [ 2580.388573] Key type ._llcrypt registered [ 2580.390260] Key type .llcrypt registered [ 2580.450045] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 2586.866939] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 2587.166510] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2587.210069] alg: No test for adler32 (adler32-zlib) [ 2588.073359] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 2588.179438] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 2589.768113] Key type lgssc registered [ 2590.128939] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2594.398966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2597.143622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2599.864242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2602.926092] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2605.835556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2605.861289] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2605.868591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2606.956303] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2606.970859] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2607.015612] Lustre: lustre-MDT0000: new disk, initializing [ 2607.047179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2607.057566] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2608.779814] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2614.104677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2614.147076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2614.205515] Lustre: 144005:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2614.247255] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2614.251032] Lustre: Skipped 1 previous similar message [ 2614.302983] Lustre: lustre-MDT0001: new disk, initializing [ 2614.341173] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2614.354873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2614.359397] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2615.862467] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2619.765410] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2621.194737] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2624.229163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2624.270465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2624.416471] Lustre: lustre-OST0000: new disk, initializing [ 2624.419552] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2624.461045] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2626.884586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2626.888652] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2626.931503] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2626.996260] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2630.784487] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2633.250727] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_module obdecho/obdecho [ 2634.162285] LustreError: 146627:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 2634.167913] LustreError: 146627:0:(obd_config.c:783:class_setup()) setup ec failed (-22) [ 2635.002524] LustreError: 146724:0:(echo_client.c:1880:echo_md_handler()) Can not resolve the path : rc = -2 [ 2639.841262] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2639.844656] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2639.849768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2642.404281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2642.407486] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2642.414593] Lustre: Skipped 1 previous similar message [ 2644.943495] LustreError: 147064:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2644.982453] Lustre: server umount lustre-MDT0000 complete [ 2646.487855] LustreError: 144785:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767386723 with bad export cookie 10097167871636375157 [ 2646.493665] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2646.495397] LustreError: 144785:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2646.598827] LustreError: 147267:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2646.602500] LustreError: 147267:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2646.656080] Lustre: server umount lustre-MDT0001 complete [ 2658.673870] LustreError: 147469:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2658.676264] LustreError: 147469:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2658.716781] Lustre: server umount lustre-OST0000 complete [ 2667.736659] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 2672.367783] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 2676.502867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2679.773444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2682.971538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2686.276514] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2689.410853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2689.436990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2689.541670] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2689.554884] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2689.598952] Lustre: lustre-MDT0000: new disk, initializing [ 2689.627846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2689.638265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2691.298892] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2696.656806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2696.685261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2696.723571] Lustre: 150411:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2696.748251] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2696.750380] Lustre: Skipped 1 previous similar message [ 2696.787351] Lustre: lustre-MDT0001: new disk, initializing [ 2696.816083] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2696.829775] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2696.834749] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2698.641738] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2702.483047] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2704.142515] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2707.371494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2707.407085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2707.549328] Lustre: lustre-OST0000: new disk, initializing [ 2707.552432] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2707.595938] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2709.323201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2709.327899] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2709.360308] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2709.936497] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2713.719342] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2715.618497] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2715.688690] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2717.306809] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2717.383172] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2719.719207] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2719.719549] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2719.727481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2719.730602] LustreError: Skipped 1 previous similar message [ 2719.736389] Lustre: Skipped 3 previous similar messages [ 2724.017143] LustreError: 153350:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2724.021226] LustreError: 153350:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2724.074513] Lustre: server umount lustre-OST0000 complete [ 2727.904747] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2727.906735] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2727.909608] Lustre: Skipped 1 previous similar message [ 2727.911086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2735.074153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2735.077673] Lustre: Skipped 3 previous similar messages [ 2740.192395] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2740.248802] LustreError: 153551:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2740.252564] LustreError: 153551:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2740.286922] Lustre: server umount lustre-MDT0000 complete [ 2742.116095] LustreError: 151192:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767386818 with bad export cookie 10097167871636376116 [ 2742.122468] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2742.123359] LustreError: 151192:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2742.275466] Lustre: server umount lustre-MDT0001 complete [ 2743.510467] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 15:47:00 (1767386820) [ 2744.003856] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 2744.736233] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 15:47:01 (1767386821) [ 2748.282503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2748.448314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2749.722282] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2753.506770] LustreError: 154426:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2753.510524] LustreError: 154426:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2753.553213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2754.843292] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2757.170922] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2758.859243] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2762.038744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2764.491512] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2766.692501] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2779.618228] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2779.622729] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2779.629857] Lustre: Skipped 1 previous similar message [ 2779.633177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2779.636904] Lustre: Skipped 2 previous similar messages [ 2785.429526] LustreError: 156693:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2785.433020] LustreError: 156693:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 2785.471524] Lustre: server umount lustre-MDT0000 complete [ 2787.116069] LustreError: 154406:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767386863 with bad export cookie 10097167871636376998 [ 2787.120149] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2787.123938] LustreError: 154406:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2787.714614] LustreError: 154422:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2788.833578] LustreError: 154422:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2788.834260] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2788.851772] Lustre: Skipped 2 previous similar messages [ 2792.833607] LustreError: 154422:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2793.322316] Lustre: server umount lustre-MDT0001 complete [ 2796.157708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2801.580585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2801.689469] LustreError: 157954:0:(llog.c:1643:llog_backup()) MGC192.168.204.106@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 2801.692722] Lustre: 157954:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.204.106@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 2816.481433] LustreError: 157962:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2816.492876] LustreError: 157962:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2826.721452] LustreError: 157980:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2826.731778] LustreError: 157980:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2844.033947] LustreError: 157979:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2844.041822] LustreError: 157979:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 2863.072312] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 2863.075239] Lustre: 157954:0:(mgc_request.c:1911:mgc_process_log()) MGC192.168.204.106@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 2877.409460] LustreError: 157979:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2877.415184] LustreError: 157979:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 19 previous similar messages [ 2889.184274] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 2889.187684] Lustre: 154417:0:(mgc_request.c:1911:mgc_process_log()) MGC192.168.204.106@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 2903.535652] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2903.540293] Lustre: Skipped 2 previous similar messages [ 2905.012770] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2906.601373] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2909.008848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2909.216324] LustreError: 154417:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 2932.199334] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 2932.204392] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x8c2058aea25fa6ff [ 2932.210652] Lustre: MGC192.168.204.106@tcp: Connection restored to 0@lo (at 0@lo) [ 2933.720490] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2936.134655] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2936.825203] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2937.836127] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2938.370350] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2940.214784] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 1 sec [ 2948.065975] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2948.067820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2948.069960] Lustre: Skipped 2 previous similar messages [ 2948.071832] Lustre: Skipped 9 previous similar messages [ 2948.074575] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2951.228555] LustreError: 159446:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2951.231250] LustreError: 159446:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2951.262256] Lustre: server umount lustre-MDT0000 complete [ 2952.933366] LustreError: 154406:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767387029 with bad export cookie 10097167871636383487 [ 2952.940494] LustreError: 154406:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2952.941166] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2953.091486] Lustre: server umount lustre-MDT0001 complete [ 2964.983307] Lustre: server umount lustre-OST0000 complete [ 2973.291888] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 2977.195296] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 2981.316153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2984.596210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2987.233068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2990.114661] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2993.052099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2993.090609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2993.199591] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2993.213785] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2993.265772] Lustre: lustre-MDT0000: new disk, initializing [ 2993.302814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2993.307195] Lustre: Skipped 1 previous similar message [ 2993.317773] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2995.034460] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 2999.913531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2999.938260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2999.981086] Lustre: 162791:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3000.104900] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3001.644330] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3005.670803] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3007.199080] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3010.302728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3010.341178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3012.884533] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3015.689820] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3015.741087] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3016.763807] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3018.491228] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3018.591776] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3020.142549] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3020.230218] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3025.891359] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3025.891692] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3025.894557] LustreError: Skipped 1 previous similar message [ 3025.900936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3025.905744] Lustre: Skipped 2 previous similar messages [ 3027.187144] LustreError: 165731:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3027.191208] LustreError: 165731:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3027.250776] Lustre: server umount lustre-OST0000 complete [ 3043.296155] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3043.382214] Lustre: server umount lustre-MDT0000 complete [ 3045.169761] LustreError: 162783:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767387121 with bad export cookie 10097167871636400315 [ 3045.175683] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3045.176726] LustreError: 162783:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3045.326892] Lustre: server umount lustre-MDT0001 complete [ 3049.428164] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 15:52:05 (1767387125) [ 3052.648653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3052.881885] LustreError: 166805:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3052.891717] LustreError: 166805:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 42 previous similar messages [ 3054.656548] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3058.467350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3060.433951] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3062.997345] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3064.628848] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3067.861216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3070.610460] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3072.929817] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3083.483279] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 15:52:39 (1767387159) [ 3084.769314] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3084.773584] LustreError: Skipped 1 previous similar message [ 3090.792446] Lustre: server umount lustre-OST0000 complete [ 3093.985535] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3093.988987] Lustre: Skipped 5 previous similar messages [ 3093.990746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3093.992529] Lustre: Skipped 10 previous similar messages [ 3098.702861] LustreError: 169449:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3098.706788] LustreError: 169449:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 3098.740575] Lustre: server umount lustre-MDT0000 complete [ 3100.445908] LustreError: 166785:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767387177 with bad export cookie 10097167871636401197 [ 3100.450321] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3100.453360] LustreError: 166785:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3100.605665] Lustre: server umount lustre-MDT0001 complete [ 3104.997508] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 3106.093834] Key type lgssc unregistered [ 3106.272633] LNet: 170181:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3106.277554] LNetError: 170181:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3106.290583] LNet: Removed LNI 192.168.204.106@tcp [ 3106.731877] Key type .llcrypt unregistered [ 3106.733855] Key type ._llcrypt unregistered [ 3115.256717] Key type ._llcrypt registered [ 3115.257887] Key type .llcrypt registered [ 3115.304199] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 3115.651381] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3115.717398] alg: No test for adler32 (adler32-zlib) [ 3116.573691] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 3116.670272] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 3118.256253] Key type lgssc registered [ 3118.779303] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3122.412806] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3122.422754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3137.888351] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3143.008224] LustreError: 171382:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.106@tcp: failed processing log, type 4: rc = -110 [ 3148.128459] LustreError: 171382:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 3148.135921] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 3148.139529] LustreError: MGC192.168.204.106@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 3148.146179] LustreError: 171382:0:(tgt_mount.c:1822:server_start_targets()) failed to start server lustre-OST0000: -22 [ 3148.151482] LustreError: 171382:0:(tgt_mount.c:2572:server_fill_super()) Unable to start targets: -22 [ 3148.156254] LustreError: 171382:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 3148.160827] LustreError: 171382:0:(obd_class.h:479:obd_check_dev()) Device 1 not setup [ 3148.195396] Lustre: server umount lustre-OST0000 complete [ 3148.198102] LustreError: 171382:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 3152.382723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3153.597334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3155.317561] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3159.346666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3159.516682] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3161.279231] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3163.838246] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3165.635346] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3168.928336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3169.024716] LustreError: 173473:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 3169.028952] Lustre: 173473:0:(mgc_request_server.c:743:mgc_llog_local_copy()) MGC192.168.204.106@tcp: invalid local config log lustre-OST0000: rc = -22 [ 3169.081994] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3171.647641] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3174.144211] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3175.205249] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3175.208368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3175.212466] Lustre: Skipped 2 previous similar messages [ 3175.215940] Lustre: Skipped 1 previous similar message [ 3180.515452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3180.518992] Lustre: Skipped 3 previous similar messages [ 3181.391254] LustreError: 174121:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3181.395314] LustreError: 174121:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3181.433520] Lustre: server umount lustre-MDT0000 complete [ 3182.965050] LustreError: 171862:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767387259 with bad export cookie 14422002182907525451 [ 3182.972229] LustreError: 171862:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3182.972435] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3183.066235] LustreError: 174326:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3183.069528] LustreError: 174326:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3183.129583] Lustre: server umount lustre-MDT0001 complete [ 3195.001275] LustreError: 174526:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3195.005251] LustreError: 174526:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3195.063908] Lustre: server umount lustre-OST0000 complete [ 3198.479850] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 3202.518198] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 3205.486474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3208.150295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3211.020832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3213.754118] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3216.692342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3216.728871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3216.849755] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3216.869193] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3216.915330] Lustre: lustre-MDT0000: new disk, initializing [ 3216.946729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3216.957120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3218.756295] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3224.256943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3224.296659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3224.345266] Lustre: 176861:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3224.376974] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3224.380341] Lustre: Skipped 1 previous similar message [ 3224.431490] Lustre: lustre-MDT0001: new disk, initializing [ 3224.470486] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3224.481810] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3224.484446] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3225.576493] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3229.221210] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3230.922210] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3234.006682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3234.038725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3234.170281] Lustre: lustre-OST0000: new disk, initializing [ 3234.173807] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3234.231341] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3234.287450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3234.291529] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3234.301180] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3236.563731] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3240.490396] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3242.448139] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3242.571900] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3244.543484] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3244.675885] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3249.639444] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3249.640801] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3249.646545] LustreError: Skipped 1 previous similar message [ 3249.657163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3249.663089] Lustre: Skipped 1 previous similar message [ 3251.699202] LustreError: 179804:0:(obd_class.h:479:obd_check_dev()) Device 20 not setup [ 3251.704058] LustreError: 179804:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3251.769169] Lustre: server umount lustre-OST0000 complete [ 3254.756556] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3254.757799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3254.763643] Lustre: Skipped 2 previous similar messages [ 3259.736308] LustreError: 180005:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3259.740260] LustreError: 180005:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3259.788834] Lustre: server umount lustre-MDT0000 complete [ 3259.873762] LustreError: 176872:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3259.883125] LustreError: 176872:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3261.496207] LustreError: 176853:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767387338 with bad export cookie 14422002182907526137 [ 3261.503312] LustreError: 176853:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3261.504710] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3261.677588] Lustre: server umount lustre-MDT0001 complete [ 3265.700190] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 15:55:42 (1767387342) [ 3273.311791] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 3274.846618] Key type lgssc unregistered [ 3275.021675] LNet: 181389:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3275.026641] LNetError: 181389:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3275.036720] LNet: Removed LNI 192.168.204.106@tcp [ 3275.518162] Key type .llcrypt unregistered [ 3275.520077] Key type ._llcrypt unregistered [ 3284.347368] Key type ._llcrypt registered [ 3284.348631] Key type .llcrypt registered [ 3284.409595] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 3284.948985] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3284.961098] alg: No test for adler32 (adler32-zlib) [ 3285.847628] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 3285.946753] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 3287.544164] Key type lgssc registered [ 3288.005220] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3291.851547] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3291.863281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3293.122746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3295.062457] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3298.830666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3298.955921] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3300.605123] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3302.765517] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3304.542580] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3308.266673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3308.378564] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3308.380351] LustreError: 184207:0:(llog.c:1684:llog_backup()) MGC192.168.204.106@tcp: failed to backup log lustre-OST0000: rc = -28 [ 3308.384309] Lustre: 184207:0:(mgc_request_server.c:740:mgc_llog_local_copy()) MGC192.168.204.106@tcp: can't backup local config lustre-OST0000: rc = -28 [ 3308.444778] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3310.662821] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3313.116756] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3314.660792] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3314.661885] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3314.666432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3314.672729] Lustre: Skipped 1 previous similar message [ 3314.679207] Lustre: Skipped 2 previous similar messages [ 3319.781384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3319.784545] Lustre: Skipped 2 previous similar messages [ 3320.344213] LustreError: 184859:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3320.380017] Lustre: server umount lustre-MDT0000 complete [ 3322.207357] LustreError: 182546:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767387398 with bad export cookie 3300458458662890700 [ 3322.214481] LustreError: 182546:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3322.216426] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3322.316473] LustreError: 185064:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3322.320447] LustreError: 185064:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3322.390408] Lustre: server umount lustre-MDT0001 complete [ 3333.496396] LustreError: 185265:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3333.500144] LustreError: 185265:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3333.571178] Lustre: server umount lustre-OST0000 complete [ 3337.862138] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 3341.342980] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 3345.648257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3348.412940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3351.508479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3354.501859] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3357.542165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3357.584235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3357.735817] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3357.756038] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3357.809480] Lustre: lustre-MDT0000: new disk, initializing [ 3357.848883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3357.860076] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3359.469787] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3364.098951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3364.130762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3364.170945] Lustre: 187599:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3364.194618] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3364.196131] Lustre: Skipped 1 previous similar message [ 3364.197810] LustreError: 188366:0:(llog.c:1684:llog_backup()) MGC192.168.204.106@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 3364.201196] LustreError: 188366:0:(llog.c:1684:llog_backup()) Skipped 1 previous similar message [ 3364.203948] Lustre: 188366:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.204.106@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 3364.214594] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3364.217212] Lustre: Skipped 1 previous similar message [ 3364.272160] Lustre: lustre-MDT0001: new disk, initializing [ 3364.322454] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3364.343822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3364.349483] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3366.159794] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3369.899205] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3371.468293] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3374.003682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3374.023933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3374.104020] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3374.105312] Lustre: Skipped 1 previous similar message [ 3374.106532] LustreError: 189327:0:(llog.c:1684:llog_backup()) MGC192.168.204.106@tcp: failed to backup log lustre-OST0000: rc = -28 [ 3374.109056] LustreError: 189327:0:(llog.c:1684:llog_backup()) Skipped 1 previous similar message [ 3374.111045] Lustre: 189327:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.204.106@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 3374.114076] Lustre: 189327:0:(mgc_request_server.c:763:mgc_llog_local_copy()) Skipped 1 previous similar message [ 3374.121578] Lustre: lustre-OST0000: new disk, initializing [ 3374.123295] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3374.149792] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3376.788521] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3380.785557] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3382.568098] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3383.306410] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3383.312639] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3383.367123] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3383.720664] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3385.495934] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3385.619770] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3388.388040] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3388.388473] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3388.392814] LustreError: Skipped 1 previous similar message [ 3388.400391] Lustre: Skipped 1 previous similar message [ 3388.401842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3388.409486] Lustre: Skipped 1 previous similar message [ 3392.627160] LustreError: 190551:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3392.631277] LustreError: 190551:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3392.688327] Lustre: server umount lustre-OST0000 complete [ 3395.041403] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3395.046315] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3395.053659] Lustre: Skipped 1 previous similar message [ 3395.057049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3398.627245] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3400.637185] LustreError: 190752:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3400.638913] LustreError: 190752:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3400.666122] Lustre: server umount lustre-MDT0000 complete [ 3401.775141] LustreError: 188381:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767387478 with bad export cookie 3300458458662891400 [ 3401.782112] LustreError: 188381:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3401.782594] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3401.921795] Lustre: server umount lustre-MDT0001 complete [ 3406.025156] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 15:58:02 (1767387482) [ 3412.519796] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 3416.891215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3417.069766] LustreError: 192320:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3417.075635] LustreError: 192320:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3417.086489] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3418.842157] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3422.177995] LustreError: 192320:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3422.738751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3424.555446] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3425.985443] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3429.179681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3431.930499] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3435.981824] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3436.014829] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3436.076056] Lustre: lustre-OST0001: new disk, initializing [ 3436.077981] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3436.343171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3436.350124] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3436.403868] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3438.706754] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3444.834768] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3448.211122] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3454.240600] Lustre: DEBUG MARKER: ADD OST3 [ 3456.545643] loop: module loaded [ 3456.564061] loop0: detected capacity change from 0 to 204800000 [ 3456.630966] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3456.635027] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3456.638845] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3456.683976] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3459.533506] loop0: detected capacity change from 0 to 204800000 [ 3461.907138] Lustre: DEBUG MARKER: STOP OST3 [ 3462.084484] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3462.130371] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3462.207969] Lustre: lustre-OST0003: new disk, initializing [ 3462.210096] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 3462.242270] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 3462.244855] Lustre: Skipped 3 previous similar messages [ 3462.765435] Lustre: Failing over lustre-OST0003 [ 3462.791802] LustreError: 196502:0:(obd_class.h:479:obd_check_dev()) Device 31 not setup [ 3462.795113] LustreError: 196502:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 3462.821044] Lustre: server umount lustre-OST0003 complete [ 3463.440409] LustreError: 194631:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3464.270124] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3467.460163] LustreError: 194631:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3467.470948] LustreError: 194631:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3468.070107] Lustre: DEBUG MARKER: START OST3 again [ 3471.321783] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3473.401850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 3473.405919] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 3473.414571] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000401 [ 3473.722360] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3488.328483] Lustre: Failing over lustre-OST0003 [ 3488.362780] LustreError: 198607:0:(obd_class.h:479:obd_check_dev()) Device 33 not setup [ 3488.366722] LustreError: 198607:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3488.397543] Lustre: server umount lustre-OST0003 complete [ 3488.739399] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3488.739505] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3488.743981] LustreError: Skipped 1 previous similar message [ 3488.752400] LustreError: 193772:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3488.764061] LustreError: 193772:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3496.936636] LustreError: 193774:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3496.945324] LustreError: 193774:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3502.060143] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3502.062065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3502.068471] Lustre: Skipped 3 previous similar messages [ 3502.072373] Lustre: Skipped 3 previous similar messages [ 3507.770914] Lustre: server umount lustre-MDT0000 complete [ 3509.539084] LustreError: 192300:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767387586 with bad export cookie 3300458458662892268 [ 3509.544245] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3509.546054] LustreError: 192300:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3512.293098] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3512.295816] Lustre: Skipped 6 previous similar messages [ 3514.338248] LustreError: 192316:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3514.347835] LustreError: 192316:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 3515.799902] Lustre: server umount lustre-MDT0001 complete [ 3517.639542] Lustre: server umount lustre-OST0000 complete [ 3519.440592] Lustre: server umount lustre-OST0001 complete [ 3522.438755] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 3526.199287] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 3531.223293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3534.740584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3538.006819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3541.254428] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3544.008052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3544.031737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3544.113889] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3544.125636] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3544.165737] Lustre: lustre-MDT0000: new disk, initializing [ 3544.189494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3544.191558] Lustre: Skipped 1 previous similar message [ 3544.197337] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3545.902540] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3551.230818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3551.266730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3551.319811] Lustre: 201664:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3551.485702] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3553.050073] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3557.061682] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3558.906587] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3561.957978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3561.984888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3562.087108] Lustre: lustre-OST0000: new disk, initializing [ 3562.088895] Lustre: Skipped 1 previous similar message [ 3562.090905] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3562.093512] Lustre: Skipped 2 previous similar messages [ 3564.034911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3564.038042] Lustre: Skipped 1 previous similar message [ 3564.041556] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3564.079641] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3564.553419] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3568.628990] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3570.452729] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3570.573206] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3572.214102] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3572.323254] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3574.245476] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3574.247586] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3574.250179] LustreError: Skipped 1 previous similar message [ 3574.257776] Lustre: Skipped 3 previous similar messages [ 3574.261694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3574.267014] Lustre: Skipped 3 previous similar messages [ 3579.315139] LustreError: 204607:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3579.319189] LustreError: 204607:0:(obd_class.h:479:obd_check_dev()) Skipped 33 previous similar messages [ 3579.384629] Lustre: server umount lustre-OST0000 complete [ 3582.432963] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3587.379955] Lustre: server umount lustre-MDT0000 complete [ 3589.065288] LustreError: 201656:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767387665 with bad export cookie 3300458458662894515 [ 3589.069435] LustreError: 201656:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3589.069603] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3589.196229] Lustre: server umount lustre-MDT0001 complete [ 3590.729180] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 16:01:06 (1767387666) [ 3598.137587] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 3602.399875] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 3606.240183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3609.319967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3612.159741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3614.966922] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3618.268882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3618.287744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3618.360910] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3618.370633] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3618.404251] Lustre: lustre-MDT0000: new disk, initializing [ 3618.429174] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3620.054498] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3625.306696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3625.346853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3625.404597] Lustre: 207866:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3625.556944] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3627.238136] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3631.296038] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3632.989930] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3636.161323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3636.205318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3638.031641] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3639.004064] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3643.098544] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3645.006933] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3645.126324] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3646.994351] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3647.099205] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3648.485338] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3648.486121] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3648.493469] Lustre: Skipped 2 previous similar messages [ 3648.496973] LustreError: Skipped 1 previous similar message [ 3648.500518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3648.504611] Lustre: Skipped 4 previous similar messages [ 3662.304179] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3662.328135] LustreError: 210806:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3662.331722] LustreError: 210806:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3662.390149] Lustre: server umount lustre-OST0000 complete [ 3670.393903] Lustre: server umount lustre-MDT0000 complete [ 3672.167153] LustreError: 208647:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767387748 with bad export cookie 3300458458662895397 [ 3672.170252] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3672.170633] LustreError: 208647:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3672.275752] Lustre: server umount lustre-MDT0001 complete [ 3675.981937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3676.183720] LustreError: 211686:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3676.223414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3676.227359] Lustre: Skipped 5 previous similar messages [ 3677.711034] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3681.687627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3683.570987] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3686.199298] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3688.036644] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3691.470454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3694.221607] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3696.350717] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3711.461223] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3711.463629] LustreError: Skipped 1 previous similar message [ 3714.989362] Lustre: server umount lustre-OST0000 complete [ 3716.578297] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3716.579475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3716.585973] Lustre: Skipped 6 previous similar messages [ 3716.589657] Lustre: Skipped 9 previous similar messages [ 3722.612082] Lustre: server umount lustre-MDT0000 complete [ 3724.236965] LustreError: 211666:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767387800 with bad export cookie 3300458458662896279 [ 3724.244450] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3724.244697] LustreError: 211666:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3724.398140] Lustre: server umount lustre-MDT0001 complete [ 3832.394222] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 3833.692329] Key type lgssc unregistered [ 3833.845669] LNet: 215019:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3833.850512] LNetError: 215019:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3833.860567] LNet: Removed LNI 192.168.204.106@tcp [ 3834.311149] Key type .llcrypt unregistered [ 3834.313376] Key type ._llcrypt unregistered [ 3844.959760] Key type ._llcrypt registered [ 3844.960975] Key type .llcrypt registered [ 3844.996743] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 3852.788589] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 3853.198254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3853.218200] alg: No test for adler32 (adler32-zlib) [ 3854.066245] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 3854.153050] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 3855.736162] Key type lgssc registered [ 3856.294164] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3860.283537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3863.291358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3866.164296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3869.229527] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3872.003735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3872.023507] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3872.028368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3873.112082] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3873.124228] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3873.170483] Lustre: lustre-MDT0000: new disk, initializing [ 3873.200811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3873.210679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3874.874303] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3879.767852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3879.807384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3879.857302] Lustre: 218319:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3879.891310] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3879.894984] Lustre: Skipped 1 previous similar message [ 3879.944369] Lustre: lustre-MDT0001: new disk, initializing [ 3879.983782] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3879.999366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3880.003240] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3881.524261] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3885.618635] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3887.494437] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3890.760598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3890.792678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3890.884833] Lustre: lustre-OST0000: new disk, initializing [ 3890.886991] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3890.914124] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3892.802633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3892.806311] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3892.850892] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3893.249128] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3897.283794] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3899.168359] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3899.285304] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3901.015089] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3901.120737] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3902.945200] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3902.949775] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3902.957961] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3903.460241] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3903.461911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3903.464918] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3908.581625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3908.585415] Lustre: Skipped 2 previous similar messages [ 3913.700436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3913.704145] Lustre: Skipped 1 previous similar message [ 3916.256152] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3916.284260] LustreError: 221257:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3916.341464] Lustre: server umount lustre-OST0000 complete [ 3918.818468] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3918.819541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3918.826024] Lustre: Skipped 1 previous similar message [ 3918.832382] Lustre: Skipped 1 previous similar message [ 3924.368856] LustreError: 221461:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3924.372824] LustreError: 221461:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3924.405319] Lustre: server umount lustre-MDT0000 complete [ 3926.172886] LustreError: 220064:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767388002 with bad export cookie 14294933810280094540 [ 3926.179110] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3926.179950] LustreError: 220064:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3926.272424] LustreError: 221664:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3926.275541] LustreError: 221664:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3926.336097] Lustre: server umount lustre-MDT0001 complete [ 3934.852907] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 3939.132241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3939.353773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3941.281719] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3944.417904] LustreError: 222836:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3944.427971] LustreError: 222836:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3945.489661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3945.649438] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3947.418561] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3948.851785] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3951.949437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3954.779215] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3958.580126] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3958.602400] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3958.649651] Lustre: lustre-OST0001: new disk, initializing [ 3958.651220] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3958.678939] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3958.680566] Lustre: Skipped 1 previous similar message [ 3961.221739] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 3965.437388] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3965.440151] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3965.470244] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3967.022666] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3971.424500] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3981.532252] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 16:07:38 (1767388058) [ 3984.530424] LustreError: 226218:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3984.534363] LustreError: 226218:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3984.572553] Lustre: server umount lustre-MDT0000 complete [ 3985.892891] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3985.900645] Lustre: Skipped 3 previous similar messages [ 3985.901264] LustreError: 222832:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3985.911686] LustreError: 222832:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3986.263962] LustreError: 223939:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767388062 with bad export cookie 14294933810280095422 [ 3986.270703] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3986.271034] LustreError: 223939:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3986.428851] Lustre: server umount lustre-MDT0001 complete [ 3998.205131] LustreError: 226620:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3998.209081] LustreError: 226620:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 3998.235586] Lustre: server umount lustre-OST0000 complete [ 4010.297591] LustreError: 226822:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4010.301570] LustreError: 226822:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4010.370182] Lustre: server umount lustre-OST0001 complete [ 4013.141138] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 4016.912380] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 4021.481278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4024.279992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4027.009075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4030.360211] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4033.625160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4033.663456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4033.779261] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4033.795681] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4033.846662] Lustre: lustre-MDT0000: new disk, initializing [ 4033.883402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4033.893815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4035.858108] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4041.509067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4041.547110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4041.606655] Lustre: 229007:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4041.777000] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4043.508792] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4047.560385] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4049.356218] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4052.824514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4052.866385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4053.043931] Lustre: lustre-OST0000: new disk, initializing [ 4053.046564] Lustre: Skipped 1 previous similar message [ 4053.050063] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4053.053450] Lustre: Skipped 2 previous similar messages [ 4054.750804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4054.756087] Lustre: Skipped 1 previous similar message [ 4054.760366] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4054.822077] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4055.862927] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4059.987446] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4062.034032] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4062.170058] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4064.249488] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4064.369602] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4070.373472] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4070.375164] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4070.381977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4070.385157] Lustre: Skipped 1 previous similar message [ 4070.390375] Lustre: Skipped 2 previous similar messages [ 4071.414180] LustreError: 231948:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4071.418128] LustreError: 231948:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4071.483297] Lustre: server umount lustre-OST0000 complete [ 4087.264138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4087.359313] Lustre: server umount lustre-MDT0000 complete [ 4088.697820] LustreError: 228998:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767388165 with bad export cookie 14294933810280097543 [ 4088.705525] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4088.879852] Lustre: server umount lustre-MDT0001 complete [ 4092.916730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4093.138046] LustreError: 232829:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4093.175153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4093.178156] Lustre: Skipped 2 previous similar messages [ 4094.856803] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4098.530434] LustreError: 232829:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4098.908610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4101.014964] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4103.555548] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4105.020460] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4108.341703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4111.118912] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4113.596448] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4256.230237] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4256.231704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4256.237904] Lustre: Skipped 3 previous similar messages [ 4256.241631] Lustre: Skipped 6 previous similar messages [ 4256.246278] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4256.251526] LustreError: Skipped 1 previous similar message [ 4270.048127] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4270.077977] LustreError: 235114:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4270.081912] LustreError: 235114:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4270.138044] Lustre: server umount lustre-OST0000 complete [ 4273.121543] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4273.126259] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4278.062379] Lustre: server umount lustre-MDT0000 complete [ 4279.842879] LustreError: 232809:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767388356 with bad export cookie 14294933810280098432 [ 4279.848334] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4279.849982] LustreError: 232809:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4280.004394] Lustre: server umount lustre-MDT0001 complete [ 4376.991665] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 4378.141566] Key type lgssc unregistered [ 4378.251320] LNet: 236087:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4378.253597] LNetError: 236087:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4378.262283] LNet: Removed LNI 192.168.204.106@tcp [ 4378.500078] Key type .llcrypt unregistered [ 4378.501188] Key type ._llcrypt unregistered [ 4387.995926] Key type ._llcrypt registered [ 4387.997490] Key type .llcrypt registered [ 4388.039149] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 4395.902556] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 4396.164487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4396.176646] alg: No test for adler32 (adler32-zlib) [ 4397.062750] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 4397.172187] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 4398.768164] Key type lgssc registered [ 4399.335655] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4403.736065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4407.001887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4410.012371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4413.140747] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4420.945656] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 4425.323058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4425.357337] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4425.366477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4426.472492] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4426.489446] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4426.543307] Lustre: lustre-MDT0000: new disk, initializing [ 4426.572218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4426.581209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4428.400210] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4433.972912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4434.009623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4434.062722] Lustre: 240084:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4434.102530] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4434.106144] Lustre: Skipped 1 previous similar message [ 4434.155937] Lustre: lustre-MDT0001: new disk, initializing [ 4434.208282] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4434.229751] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4434.235968] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4435.900451] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4438.720624] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4441.889360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4441.929963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4442.046535] Lustre: lustre-OST0000: new disk, initializing [ 4442.048544] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4442.083129] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4443.521205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4443.525883] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4443.581485] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4444.344693] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4449.570816] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4449.611816] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4449.706139] Lustre: lustre-OST0001: new disk, initializing [ 4449.709772] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4449.766640] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4450.396136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4450.402144] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4450.450213] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4452.284794] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4458.774695] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4465.230720] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4475.361542] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4475.367504] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4475.379669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4476.385411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4476.388268] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4476.391450] Lustre: Skipped 1 previous similar message [ 4476.550773] LustreError: 243610:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4476.579493] Lustre: server umount lustre-MDT0000 complete [ 4478.212846] LustreError: 240076:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767388554 with bad export cookie 18213836111723479130 [ 4478.219299] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4478.219836] LustreError: 240076:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4480.342662] LustreError: 243812:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 4480.346346] LustreError: 243812:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4480.410946] Lustre: server umount lustre-MDT0001 complete [ 4491.517170] LustreError: 244012:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4491.521529] LustreError: 244012:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4491.561749] Lustre: server umount lustre-OST0000 complete [ 4503.419394] LustreError: 244214:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4503.423334] LustreError: 244214:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4503.494639] Lustre: server umount lustre-OST0001 complete [ 4509.871436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4513.831098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4514.290801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4518.119185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4522.238129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4522.700327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4531.429564] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 4536.118801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4536.133457] Lustre: lustre-MDT0000: reset Object Index mappings [ 4536.369120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4538.055052] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4541.410129] LustreError: 247521:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4541.419165] LustreError: 247521:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4542.136979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4542.145866] Lustre: lustre-MDT0001: reset Object Index mappings [ 4544.052361] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4545.492415] Lustre: 248633:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4548.403747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4548.564540] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4548.568071] Lustre: Skipped 1 previous similar message [ 4551.239282] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4551.658339] LustreError: 248987:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4555.342438] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4557.925991] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4560.627770] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 4560.629095] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4567.365086] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4568.981960] Lustre: 250478:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4576.230463] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4576.232880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4576.235981] Lustre: Skipped 3 previous similar messages [ 4576.237668] Lustre: Skipped 4 previous similar messages [ 4581.346579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4581.349326] Lustre: Skipped 3 previous similar messages [ 4581.585682] LustreError: 250583:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 4581.589421] LustreError: 250583:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4581.628243] Lustre: server umount lustre-MDT0000 complete [ 4583.346782] LustreError: 248249:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767388659 with bad export cookie 18213836111723485906 [ 4583.353964] LustreError: 248249:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4583.355267] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4583.501254] Lustre: server umount lustre-MDT0001 complete [ 4594.676087] LustreError: 250988:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 4594.678097] LustreError: 250988:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 4594.693334] Lustre: server umount lustre-OST0000 complete [ 4605.629700] Lustre: server umount lustre-OST0001 complete [ 4615.226437] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 16:18:11 (1767388691) [ 4622.887774] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 4627.556806] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 4631.995388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4635.194768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4638.330154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4641.647712] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4645.061956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4645.101519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4645.217197] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4645.232808] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4645.285738] Lustre: lustre-MDT0000: new disk, initializing [ 4645.322728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4645.326495] Lustre: Skipped 1 previous similar message [ 4645.336847] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4647.190922] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4652.852269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4652.898500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4652.955285] Lustre: 254708:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4653.135943] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4654.995884] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4659.167363] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4660.976559] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4664.361658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4664.411270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4664.558923] Lustre: lustre-OST0000: new disk, initializing [ 4664.561928] Lustre: Skipped 1 previous similar message [ 4664.565344] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4664.568542] Lustre: Skipped 2 previous similar messages [ 4665.816758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4665.821049] Lustre: Skipped 1 previous similar message [ 4665.824428] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4665.883620] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4667.389945] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4671.556614] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4673.561564] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4673.692425] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4675.588598] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4675.714246] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4681.192797] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4681.195448] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4681.197540] LustreError: Skipped 1 previous similar message [ 4681.205098] Lustre: Skipped 1 previous similar message [ 4681.209089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4681.213717] Lustre: Skipped 1 previous similar message [ 4682.738224] LustreError: 257643:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4682.741849] LustreError: 257643:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 4682.806982] Lustre: server umount lustre-OST0000 complete [ 4686.308619] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4686.315513] Lustre: Skipped 1 previous similar message [ 4690.817822] Lustre: server umount lustre-MDT0000 complete [ 4691.426331] LustreError: 255653:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4691.433316] LustreError: 255653:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4692.533505] LustreError: 257848:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767388769 with bad export cookie 18213836111723487999 [ 4692.537900] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4692.540654] LustreError: 257848:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4692.692667] Lustre: server umount lustre-MDT0001 complete [ 4701.473659] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 4706.348424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4706.586581] LustreError: 259222:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4706.614157] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4706.617850] Lustre: Skipped 2 previous similar messages [ 4708.616540] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4711.908547] LustreError: 259222:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4712.974806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4715.189993] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4716.672468] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4719.998407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4722.984311] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4727.180651] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4727.226143] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4727.333916] Lustre: lustre-OST0001: new disk, initializing [ 4727.336949] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4728.475912] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4728.482278] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4728.534008] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4730.086768] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4735.690560] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4737.484660] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4744.864021] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 4744.869753] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 4747.234897] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4747.238166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4747.239160] Lustre: Skipped 2 previous similar messages [ 4747.241246] Lustre: Skipped 3 previous similar messages [ 4753.355504] LustreError: 262729:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4753.358393] LustreError: 262729:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4753.400420] Lustre: server umount lustre-MDT0000 complete [ 4755.116788] LustreError: 259201:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767388831 with bad export cookie 18213836111723488881 [ 4755.123414] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4755.123845] LustreError: 259201:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4755.278889] Lustre: server umount lustre-MDT0001 complete [ 4766.739190] Lustre: server umount lustre-OST0000 complete [ 4777.599091] Lustre: server umount lustre-OST0001 complete [ 4785.371679] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 4790.245738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4790.454787] LustreError: 264508:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4790.490563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4790.493452] Lustre: Skipped 3 previous similar messages [ 4792.163352] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4795.999064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4798.163283] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4799.640235] Lustre: 265616:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4802.792858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4805.522385] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4809.832188] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4812.618143] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4815.983249] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 4816.003285] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 4821.991682] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4823.858712] Lustre: 267470:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4831.523706] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 4831.532283] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 4835.933097] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 16:21:52 (1767388912) [ 4852.200681] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4852.202652] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4852.208236] Lustre: Skipped 4 previous similar messages [ 4852.214547] Lustre: Skipped 9 previous similar messages [ 4854.035723] LustreError: 268178:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 4854.039689] LustreError: 268178:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 4854.074566] Lustre: server umount lustre-MDT0000 complete [ 4855.723838] LustreError: 265234:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767388932 with bad export cookie 18213836111723495153 [ 4855.730573] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4855.730963] LustreError: 265234:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4857.313764] LustreError: 264502:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4857.323510] LustreError: 264502:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4870.112247] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4870.232358] Lustre: server umount lustre-MDT0001 complete [ 4878.154418] Lustre: server umount lustre-OST0000 complete [ 4885.740511] Lustre: server umount lustre-OST0001 complete [ 4888.565975] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 4892.353907] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 4897.200138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4900.237210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4903.400254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4906.763666] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4914.806353] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 4919.557862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4919.586081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4919.672670] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4919.687716] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4919.746792] Lustre: lustre-MDT0000: new disk, initializing [ 4919.786252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4919.790112] Lustre: Skipped 3 previous similar messages [ 4919.800505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4921.584135] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4926.830770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4926.863850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4926.919553] Lustre: 271668:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4927.067712] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4928.767336] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4934.689169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4934.724579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4936.653382] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4937.562947] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4943.099678] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4943.128416] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4944.413843] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4945.963719] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 4951.539808] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4953.321836] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4953.325341] Lustre: Skipped 1 previous similar message [ 4964.268045] Lustre: DEBUG MARKER: == conf-sanity test 160: MGC updates failnodes from all participants ========================================================== 16:24:00 (1767389040) [ 4967.905529] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4967.908433] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4967.912838] Lustre: Skipped 2 previous similar messages [ 4967.915953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4967.919053] Lustre: Skipped 6 previous similar messages [ 4971.460826] Lustre: server umount lustre-MDT0000 complete [ 4973.186973] LustreError: 271660:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767389049 with bad export cookie 18213836111723503014 [ 4973.196616] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4973.350178] Lustre: server umount lustre-MDT0001 complete [ 4985.403163] LustreError: 275680:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4985.404750] LustreError: 275680:0:(obd_class.h:479:obd_check_dev()) Skipped 41 previous similar messages [ 4985.432051] Lustre: server umount lustre-OST0000 complete [ 4997.246395] Lustre: server umount lustre-OST0001 complete [ 5005.464324] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 5009.554386] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 5013.929370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5016.935043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5019.806311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5022.787597] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5026.246462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5026.284095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5026.412407] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5026.430907] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5026.434627] Lustre: Skipped 4 previous similar messages [ 5026.489550] Lustre: lustre-MDT0000: new disk, initializing [ 5026.491166] Lustre: Skipped 3 previous similar messages [ 5026.535104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5026.538791] Lustre: Skipped 3 previous similar messages [ 5028.472518] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5034.027777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5034.075231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5034.131097] Lustre: 278723:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5034.303160] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5034.308485] Lustre: Skipped 2 previous similar messages [ 5035.987453] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5040.140176] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5042.026293] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5045.227091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5045.269278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5047.316318] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5048.163496] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5052.110415] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5053.129183] Lustre: Mounted lustre-client [ 5053.537555] LustreError: 281226:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff9a2de0029800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 5053.576198] Lustre: Unmounted lustre-client [ 5058.535627] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5058.541479] Lustre: Skipped 3 previous similar messages [ 5058.544506] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5058.547985] Lustre: Skipped 5 previous similar messages [ 5060.083266] Lustre: server umount lustre-OST0000 complete [ 5068.100673] Lustre: server umount lustre-MDT0000 complete [ 5068.770405] LustreError: 278733:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5068.779349] LustreError: 278733:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5069.930214] LustreError: 279502:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767389146 with bad export cookie 18213836111723504540 [ 5069.937313] LustreError: 279502:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 5069.937825] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5070.117766] Lustre: server umount lustre-MDT0001 complete [ 5075.068358] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 5076.566657] Key type lgssc unregistered [ 5103.712149] LNet: 1 local NIs in recovery (showing 1): 192.168.204.106@tcp [ 5104.416592] LNet: 282502:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5104.421371] LNetError: 282502:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5183.270690] LNet: Removed LNI 192.168.204.106@tcp [ 5183.761171] Key type .llcrypt unregistered [ 5183.763189] Key type ._llcrypt unregistered [ 5192.255209] Key type ._llcrypt registered [ 5192.256500] Key type .llcrypt registered [ 5192.312066] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 5200.601533] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 5200.944955] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5200.979076] alg: No test for adler32 (adler32-zlib) [ 5201.870448] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 5201.993641] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 5203.600186] Key type lgssc registered [ 5204.205326] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5208.039281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5211.232349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5214.338753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5217.311843] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5218.306239] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 16:28:14 (1767389294) [ 5221.286191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5221.325093] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5221.333930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5222.438225] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5222.453200] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5222.490349] Lustre: lustre-MDT0000: new disk, initializing [ 5222.521270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5222.527350] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5224.132155] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5229.407648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5229.445595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5229.499356] Lustre: 285674:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5229.540392] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5229.544147] Lustre: Skipped 1 previous similar message [ 5229.599244] Lustre: lustre-MDT0001: new disk, initializing [ 5229.653472] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5229.675611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5229.683282] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5231.420061] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5235.044308] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5236.765527] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5239.629694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5239.655819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5239.778810] Lustre: lustre-OST0000: new disk, initializing [ 5239.781200] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5239.816323] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5242.322953] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_default_debug -1 all [ 5245.957794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5245.963968] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5246.018096] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5246.354915] Lustre: DEBUG MARKER: oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5256.165568] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5256.167707] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5256.168494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5256.169353] LustreError: Skipped 1 previous similar message [ 5256.175100] Lustre: Skipped 1 previous similar message [ 5261.285410] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5261.289283] Lustre: Skipped 2 previous similar messages [ 5265.376146] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5265.401150] LustreError: 288119:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5265.458531] Lustre: server umount lustre-OST0000 complete [ 5270.497562] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5270.502426] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5270.510809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5271.523743] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5273.243496] LustreError: 288321:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5273.246988] LustreError: 288321:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5273.276447] Lustre: server umount lustre-MDT0000 complete [ 5275.014432] LustreError: 287423:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767389351 with bad export cookie 5221368625754098601 [ 5275.017429] LustreError: MGC192.168.204.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5275.018386] LustreError: 287423:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5275.086397] LustreError: 288523:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5275.088421] LustreError: 288523:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5275.143135] Lustre: server umount lustre-MDT0001 complete [ 5279.861780] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 5281.286760] Key type lgssc unregistered [ 5281.456469] LNet: 289054:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5281.461191] LNetError: 289054:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5281.472526] LNet: Removed LNI 192.168.204.106@tcp [ 5281.926475] Key type .llcrypt unregistered [ 5281.928761] Key type ._llcrypt unregistered [ 5323.892410] Key type ._llcrypt registered [ 5323.894278] Key type .llcrypt registered [ 5323.953898] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 5324.125348] Key type .llcrypt unregistered [ 5324.127758] Key type ._llcrypt unregistered [ 5360.306920] Key type ._llcrypt registered [ 5360.308561] Key type .llcrypt registered [ 5360.361664] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 5360.514345] Key type .llcrypt unregistered [ 5360.516500] Key type ._llcrypt unregistered [ 5374.362478] Key type ._llcrypt registered [ 5374.364604] Key type .llcrypt registered [ 5374.427752] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 5374.590280] Key type .llcrypt unregistered [ 5374.592390] Key type ._llcrypt unregistered [ 5389.071056] Key type ._llcrypt registered [ 5389.072186] Key type .llcrypt registered [ 5389.119214] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 5389.275494] Key type .llcrypt unregistered [ 5389.277523] Key type ._llcrypt unregistered [ 5427.693525] Key type ._llcrypt registered [ 5427.694690] Key type .llcrypt registered [ 5427.747670] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 5427.920335] Key type .llcrypt unregistered [ 5427.922489] Key type ._llcrypt unregistered [ 5443.225814] Key type ._llcrypt registered [ 5443.227495] Key type .llcrypt registered [ 5443.293862] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 5443.459378] Key type .llcrypt unregistered [ 5443.461501] Key type ._llcrypt unregistered [ 5458.687375] Key type ._llcrypt registered [ 5458.688833] Key type .llcrypt registered [ 5458.752819] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 5458.925318] Key type .llcrypt unregistered [ 5458.927315] Key type ._llcrypt unregistered [ 5474.234809] Key type ._llcrypt registered [ 5474.236381] Key type .llcrypt registered [ 5474.298967] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 5474.468299] Key type .llcrypt unregistered [ 5474.470416] Key type ._llcrypt unregistered [ 5489.849926] Key type ._llcrypt registered [ 5489.851476] Key type .llcrypt registered [ 5489.922637] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 5490.090531] Key type .llcrypt unregistered [ 5490.093128] Key type ._llcrypt unregistered [ 5505.171815] Key type ._llcrypt registered [ 5505.173323] Key type .llcrypt registered [ 5505.239691] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 5505.415347] Key type .llcrypt unregistered [ 5505.417445] Key type ._llcrypt unregistered [ 5535.461704] Key type ._llcrypt registered [ 5535.463243] Key type .llcrypt registered [ 5535.522354] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 5535.697323] Key type .llcrypt unregistered [ 5535.699445] Key type ._llcrypt unregistered [ 5550.640430] Key type ._llcrypt registered [ 5550.642989] Key type .llcrypt registered [ 5550.707643] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 5550.874604] Key type .llcrypt unregistered [ 5550.876439] Key type ._llcrypt unregistered [ 5565.798850] Key type ._llcrypt registered [ 5565.800150] Key type .llcrypt registered [ 5565.855249] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 5566.025353] Key type .llcrypt unregistered [ 5566.027708] Key type ._llcrypt unregistered [ 5593.252947] Key type ._llcrypt registered [ 5593.254734] Key type .llcrypt registered [ 5593.308327] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing set_hostid [ 5602.020784] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing load_modules_local [ 5602.374357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5602.428762] alg: No test for adler32 (adler32-zlib) [ 5603.338357] Lustre: Lustre: Build Version: 2.17.0_RC4_2_g45eb11f [ 5603.455934] LNet: Added LNI 192.168.204.106@tcp [8/256/0/180] [ 5605.056154] Key type lgssc registered [ 5605.626488] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5610.338176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5613.749767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5616.900804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5620.223420] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5621.151640] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 5499 sec ========== 16:34:57 (1767389697) [ 5622.034359] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 16:34:58 (1767389698) === [ 5623.730673] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 16:35:00 (1767389700) === [ 5643.252584] Lustre: DEBUG MARKER: oleg406-server.virtnet: executing unload_modules_local [ 5644.841455] Key type lgssc unregistered [ 5645.022824] LNet: 310763:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5645.027498] LNetError: 310763:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5645.043644] LNet: Removed LNI 192.168.204.106@tcp [ 5645.513577] Key type .llcrypt unregistered [ 5645.515748] Key type ._llcrypt unregistered