[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 479598313 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003333] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.008476] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010017] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011013] pid_max: default: 32768 minimum: 301 [ 0.012132] LSM: Security Framework initializing [ 0.013059] Yama: becoming mindful. [ 0.014055] SELinux: Initializing. [ 0.015090] *** VALIDATE selinux *** [ 0.023834] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028555] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029156] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030111] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032026] *** VALIDATE tmpfs *** [ 0.033348] *** VALIDATE proc *** [ 0.034301] *** VALIDATE cgroup *** [ 0.035019] *** VALIDATE cgroup2 *** [ 0.037333] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038214] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040028] Spectre V2 : User space: Vulnerable [ 0.041009] Speculative Store Bypass: Vulnerable [ 0.044920] debug: unmapping init [mem 0xffffffff91e59000-0xffffffff91e60fff] [ 0.047193] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048744] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049038] ... version: 2 [ 0.050009] ... bit width: 48 [ 0.050989] ... generic registers: 4 [ 0.051010] ... value mask: 0000ffffffffffff [ 0.052026] ... max period: 00007fffffffffff [ 0.053027] ... fixed-purpose events: 3 [ 0.054020] ... event mask: 000000070000000f [ 0.056307] rcu: Hierarchical SRCU implementation. [ 0.058783] smp: Bringing up secondary CPUs ... [ 0.059685] x86: Booting SMP configuration: [ 0.060036] .... node #0, CPUs: #1 #2 #3 [ 0.063585] smp: Brought up 1 node, 4 CPUs [ 0.065018] smpboot: Max logical packages: 1 [ 0.066028] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.240027] node 0 deferred pages initialised in 172ms [ 0.244177] devtmpfs: initialized [ 0.247738] x86/mm: Memory block size: 128MB [ 0.251992] gcov: version magic: 0x41383552 [ 0.255176] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.259112] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.261272] pinctrl core: initialized pinctrl subsystem [ 0.263124] [ 0.263818] ************************************************************* [ 0.266013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.268010] ** ** [ 0.271010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.272009] ** ** [ 0.274010] ** This means that this kernel is built to expose internal ** [ 0.277016] ** IOMMU data structures, which may compromise security on ** [ 0.279012] ** your system. ** [ 0.282011] ** ** [ 0.284009] ** If you see this message and you are not debugging the ** [ 0.286011] ** kernel, report this immediately to your vendor! ** [ 0.288012] ** ** [ 0.289007] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.292019] ************************************************************* [ 0.294623] NET: Registered protocol family 16 [ 0.296356] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.298056] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.300061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.304062] cpuidle: using governor menu [ 0.305808] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.306988] PCI: Using configuration type 1 for base access [ 0.307164] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.315064] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.317020] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.320176] cryptd: max_cpu_qlen set to 1000 [ 0.322248] ACPI: Added _OSI(Module Device) [ 0.324015] ACPI: Added _OSI(Processor Device) [ 0.326026] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.328021] ACPI: Added _OSI(Processor Aggregator Device) [ 0.333014] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.339425] ACPI: Interpreter enabled [ 0.340677] ACPI: PM: (supports S0 S3 S4 S5) [ 0.342042] ACPI: Using IOAPIC for interrupt routing [ 0.344100] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.347830] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.358192] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.360039] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.362018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.365091] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.369383] acpiphp: Slot [2] registered [ 0.371126] acpiphp: Slot [5] registered [ 0.372114] acpiphp: Slot [6] registered [ 0.374101] acpiphp: Slot [7] registered [ 0.375103] acpiphp: Slot [8] registered [ 0.376125] acpiphp: Slot [9] registered [ 0.378144] acpiphp: Slot [10] registered [ 0.380114] acpiphp: Slot [3] registered [ 0.381117] acpiphp: Slot [4] registered [ 0.382137] acpiphp: Slot [11] registered [ 0.384113] acpiphp: Slot [12] registered [ 0.385102] acpiphp: Slot [13] registered [ 0.387119] acpiphp: Slot [14] registered [ 0.389113] acpiphp: Slot [15] registered [ 0.390109] acpiphp: Slot [16] registered [ 0.392114] acpiphp: Slot [17] registered [ 0.394125] acpiphp: Slot [18] registered [ 0.395170] acpiphp: Slot [19] registered [ 0.397107] acpiphp: Slot [20] registered [ 0.399132] acpiphp: Slot [21] registered [ 0.401107] acpiphp: Slot [22] registered [ 0.402100] acpiphp: Slot [23] registered [ 0.404206] acpiphp: Slot [24] registered [ 0.406088] acpiphp: Slot [25] registered [ 0.408155] acpiphp: Slot [26] registered [ 0.409100] acpiphp: Slot [27] registered [ 0.411104] acpiphp: Slot [28] registered [ 0.413157] acpiphp: Slot [29] registered [ 0.414089] acpiphp: Slot [30] registered [ 0.416178] acpiphp: Slot [31] registered [ 0.418070] PCI host bridge to bus 0000:00 [ 0.420019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.424026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.426023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.429027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.432026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.435028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.437138] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.440024] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.443297] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.451635] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.455054] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.459021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.461013] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.463017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.464398] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.467702] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.470052] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.474564] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.479013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.492015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.499015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.506220] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.512018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.518019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.540015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.547814] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.555013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.562015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.580017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.591830] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.598022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.609022] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.628019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.641247] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.648014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.654017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.671015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.680485] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.689015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.695013] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.710024] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.719000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.727015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.733016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.747027] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.759893] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.761236] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.763271] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.764291] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.766222] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.771042] iommu: Default domain type: Passthrough [ 0.773423] SCSI subsystem initialized [ 0.775132] ACPI: bus type USB registered [ 0.776124] usbcore: registered new interface driver usbfs [ 0.779086] usbcore: registered new interface driver hub [ 0.780080] usbcore: registered new device driver usb [ 0.782222] pps_core: LinuxPPS API ver. 1 registered [ 0.784012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.787067] PTP clock support registered [ 0.790072] EDAC MC: Ver: 3.0.0 [ 0.792050] PCI: Using ACPI for IRQ routing [ 0.794524] NetLabel: Initializing [ 0.796015] NetLabel: domain hash size = 128 [ 0.798015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.799063] NetLabel: unlabeled traffic allowed by default [ 0.802049] vgaarb: loaded [ 0.804009] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.805013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.809172] clocksource: Switched to clocksource kvm-clock [ 0.921484] VFS: Disk quotas dquot_6.6.0 [ 0.923845] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.926668] *** VALIDATE ramfs *** [ 0.928105] *** VALIDATE hugetlbfs *** [ 0.930149] pnp: PnP ACPI init [ 0.932938] pnp: PnP ACPI: found 6 devices [ 0.951986] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.955928] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.958544] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.961572] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.964695] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.967663] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.971180] NET: Registered protocol family 2 [ 0.974122] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.979940] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.984204] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.990115] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.993953] TCP: Hash tables configured (established 65536 bind 65536) [ 0.997333] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.000822] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.004020] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.007470] NET: Registered protocol family 1 [ 1.010178] RPC: Registered named UNIX socket transport module. [ 1.012534] RPC: Registered udp transport module. [ 1.014197] RPC: Registered tcp transport module. [ 1.015861] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.018273] NET: Registered protocol family 44 [ 1.020409] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.022939] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.025301] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.028165] PCI: CLS 0 bytes, default 64 [ 1.030265] Unpacking initramfs... [ 2.520670] debug: unmapping init [mem 0xffff99133cc54000-0xffff99133ffbffff] [ 2.524937] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.526660] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.528932] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.010023] Initialise system trusted keyrings [ 3.011553] Key type blacklist registered [ 3.013577] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.021780] zbud: loaded [ 3.025418] *** VALIDATE nfs *** [ 3.026935] *** VALIDATE nfs4 *** [ 3.028427] pstore: using deflate compression [ 3.031276] Platform Keyring initialized [ 3.131803] NET: Registered protocol family 38 [ 3.133367] Key type asymmetric registered [ 3.134751] Asymmetric key parser 'x509' registered [ 3.136343] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.139247] io scheduler mq-deadline registered [ 3.141424] io scheduler kyber registered [ 3.143405] io scheduler bfq registered [ 3.145580] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.148840] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.153230] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.156808] ACPI: Power Button [PWRF] [ 3.162734] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.170434] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.183089] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.194191] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.214783] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.242888] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.272529] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.277799] Non-volatile memory driver v1.3 [ 3.279307] Linux agpgart interface v0.103 [ 3.312466] virtio_blk virtio1: [vda] 146152 512-byte logical blocks (74.8 MB/71.4 MiB) [ 3.314791] vda: detected capacity change from 0 to 74829824 [ 3.329774] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.333277] vdb: detected capacity change from 0 to 1073741824 [ 3.351814] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.355721] vdc: detected capacity change from 0 to 2621440000 [ 3.372784] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.376479] vdd: detected capacity change from 0 to 2621440000 [ 3.393502] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.396667] vde: detected capacity change from 0 to 4294967296 [ 3.413633] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.416751] vdf: detected capacity change from 0 to 4294967296 [ 3.425169] libphy: Fixed MDIO Bus: probed [ 3.431432] usbcore: registered new interface driver usbserial_generic [ 3.434695] usbserial: USB Serial support registered for generic [ 3.437174] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.442557] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.444201] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.447173] mousedev: PS/2 mouse device common for all mice [ 3.450518] rtc_cmos 00:05: RTC can wake from S4 [ 3.451192] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.454959] rtc_cmos 00:05: registered as rtc0 [ 3.458261] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.460293] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.462235] intel_pstate: CPU model not supported [ 3.468426] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.470266] hid: raw HID events driver (C) Jiri Kosina [ 3.474235] usbcore: registered new interface driver usbhid [ 3.476219] usbhid: USB HID core driver [ 3.477469] drop_monitor: Initializing network drop monitor service [ 3.478914] Initializing XFRM netlink socket [ 3.480228] NET: Registered protocol family 10 [ 3.482408] Segment Routing with IPv6 [ 3.483449] NET: Registered protocol family 17 [ 3.486166] mpls_gso: MPLS GSO support [ 3.491598] RAS: Correctable Errors collector initialized. [ 3.494661] AVX version of gcm_enc/dec engaged. [ 3.496712] AES CTR mode by8 optimization enabled [ 3.582134] sched_clock: Marking stable (3582076610, 0)->(4517555804, -935479194) [ 3.586190] registered taskstats version 1 [ 3.588268] Loading compiled-in X.509 certificates [ 3.590977] zswap: loaded using pool lzo/zbud [ 3.619897] Key type big_key registered [ 3.632698] Key type encrypted registered [ 3.634332] ima: No TPM chip found, activating TPM-bypass! [ 3.636664] ima: Allocated hash algorithm: sha1 [ 3.638765] ima: No architecture policies found [ 3.640735] evm: Initialising EVM extended attributes: [ 3.642745] evm: security.selinux [ 3.644215] evm: security.ima [ 3.645314] evm: security.capability [ 3.646797] evm: HMAC attrs: 0x1 [ 3.649284] rtc_cmos 00:05: setting system clock to 2026-08-24 13:52:43 UTC (1787579563) [ 3.656293] debug: unmapping init [mem 0xffffffff92e03000-0xffffffff92ffffff] [ 3.659563] debug: unmapping init [mem 0xffffffff91b82000-0xffffffff91e58fff] [ 3.668109] Write protecting the kernel read-only data: 28672k [ 3.671438] debug: unmapping init [mem 0xffffffff90203000-0xffffffff903fffff] [ 3.674132] debug: unmapping init [mem 0xffffffff90b14000-0xffffffff90bfffff] [ 3.710392] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.720692] systemd[1]: Detected virtualization kvm. [ 3.723151] systemd[1]: Detected architecture x86-64. [ 3.725328] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.751756] systemd[1]: No hostname configured. [ 3.753829] systemd[1]: Set hostname to . [ 3.756396] random: systemd: uninitialized urandom read (16 bytes read) [ 3.759275] systemd[1]: Initializing machine ID from random generator. [ 3.797359] random: ln: uninitialized urandom read (6 bytes read) [ 3.907915] random: systemd: uninitialized urandom read (16 bytes read) [ 3.910950] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.916276] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.920639] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.587637] device-mapper: uevent: version 1.0.3 [ 4.589621] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.424239] random: fast init done [ 5.466816] virtio_net virtio0 ens2: renamed from eth0 [ 5.542389] scsi host0: ata_piix [ 5.564089] scsi host1: ata_piix [ 5.565934] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.568789] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.284301] dracut-initqueue[580]: RTNETLINK answers: File exists [ 10.152306] random: crng init done [ 10.155558] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.689548] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.957710] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.232312] SELinux: Disabled at runtime. [ 12.292954] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.303064] systemd[1]: Detected virtualization kvm. [ 12.304727] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.858349] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.862127] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.870733] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.874893] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.878214] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.886518] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.892424] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Control Socket. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Switch Root. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-getty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice User and Session Slice. [ OK ] Reached target rpc_pipefs.target. [ 12.949919] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Huge Pages File System... Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Apply Kernel Variables... Mounting Kernel Debug File System... [ OK ] Reached target Slices. Starting udev Coldplug all Devices... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 13.412117] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.810772] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.833478] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.013327] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.028251] EDAC sbridge: Ver: 1.1.2 [ 16.023722] Key type dns_resolver registered [ 16.377253] NFS: Registering the id_resolver key type [ 16.379802] Key type id_resolver registered [ 16.382947] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg304-server login: [ 43.807311] libcfs: loading out-of-tree module taints kernel. [ 43.829579] Key type ._llcrypt registered [ 43.831473] Key type .llcrypt registered [ 43.884804] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 52.701395] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 53.491480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 53.501241] alg: No test for adler32 (adler32-zlib) [ 54.552378] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 54.929284] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 56.584566] Key type lgssc registered [ 58.805864] Lustre: Echo OBD driver; http://www.lustre.org/ [ 76.327981] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 94.025154] hrtimer: interrupt took 6144667 ns [ 129.377945] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 144.723587] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 144.795453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 146.242134] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 146.290772] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 146.444211] Lustre: lustre-MDT0000: new disk, initializing [ 146.521137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 146.543711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 150.982636] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 167.131509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 167.288100] Lustre: 6480:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 167.355523] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 167.371901] Lustre: Skipped 1 previous similar message [ 167.525316] Lustre: lustre-MDT0001: new disk, initializing [ 167.620482] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 167.679413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 167.711430] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 173.840522] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 179.229922] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 189.359512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 189.715698] Lustre: lustre-OST0000: new disk, initializing [ 189.722469] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 189.739690] Lustre: 8417:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 189.841439] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 196.666921] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 198.683370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 198.704435] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 198.784711] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 210.865445] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 211.061484] Lustre: lustre-OST0001: new disk, initializing [ 211.071763] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 211.078779] Lustre: 9492:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 211.176416] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 217.201221] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 220.216557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 220.232590] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 220.285962] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 228.975995] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 234.542824] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 241.366032] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing check_logdir /tmp/testlogs/ [ 246.050771] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing yml_node [ 250.792745] Lustre: DEBUG MARKER: Client: 2.17.57.80 [ 253.848086] Lustre: DEBUG MARKER: MDS: 2.17.57.80 [ 256.887312] Lustre: DEBUG MARKER: OSS: 2.17.57.80 [ 258.967102] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 24 09:56:56 EDT 2026 [ 277.627603] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 279.587322] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 296.928857] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 296.930867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 296.939144] Lustre: Skipped 1 previous similar message [ 296.961163] Lustre: Skipped 3 previous similar messages [ 302.049520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 302.067864] Lustre: Skipped 3 previous similar messages [ 303.360424] Lustre: server umount lustre-MDT0000 complete [ 307.396708] LustreError: 6473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787579867 with bad export cookie 9166560507614044880 [ 307.398615] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 307.409384] LustreError: 6473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 312.292652] LustreError: 6487:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 312.293667] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 312.307533] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 312.309035] LustreError: 6487:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 312.366329] Lustre: Skipped 3 previous similar messages [ 314.163308] Lustre: server umount lustre-MDT0001 complete [ 318.425766] Lustre: server umount lustre-OST0000 complete [ 322.713661] Lustre: server umount lustre-OST0001 complete [ 329.940543] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 339.248733] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 380.118051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 380.416758] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 380.442659] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 380.556594] Lustre: lustre-MDT0000: new disk, initializing [ 380.668676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 380.695574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 384.747266] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 396.518832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 396.660462] Lustre: 15050:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 396.728592] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 396.732172] Lustre: Skipped 1 previous similar message [ 396.786762] Lustre: lustre-MDT0001: new disk, initializing [ 396.878919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 396.907108] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 396.919692] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 401.761728] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 412.080777] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 419.126739] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 426.827978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 427.148354] Lustre: lustre-OST0000: new disk, initializing [ 427.153916] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 427.160346] Lustre: 17017:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 427.259717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 428.806790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 428.825308] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 428.966055] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 433.788794] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 443.609612] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 448.093819] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 448.378666] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 453.248046] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 453.485768] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 459.755444] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 459.758383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 459.794856] Lustre: Skipped 1 previous similar message [ 459.805067] Lustre: Skipped 1 previous similar message [ 459.823840] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 461.634202] Lustre: server umount lustre-OST0000 complete [ 468.449865] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 468.460732] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 468.484025] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 471.729196] Lustre: server umount lustre-MDT0000 complete [ 475.110974] LustreError: 17105:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 475.133036] LustreError: 17105:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 475.546972] LustreError: 15040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787580035 with bad export cookie 9166560507614046420 [ 475.556263] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 475.567406] LustreError: 15040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 475.946823] Lustre: server umount lustre-MDT0001 complete [ 482.328932] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 10:00:40 (1787580040) [ 501.387040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 501.946675] LustreError: 19906:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 502.088479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 506.802199] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 507.362879] LustreError: 19907:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 507.384659] LustreError: 19907:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 511.460674] LustreError: 19906:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 516.308524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 521.165907] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 529.942086] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 537.331752] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 539.689337] Lustre: Failing over lustre-MDT0000 [ 539.699074] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 539.704186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 539.705888] Lustre: Skipped 2 previous similar messages [ 539.722022] Lustre: Skipped 3 previous similar messages [ 540.029591] Lustre: server umount lustre-MDT0000 complete [ 541.792734] LustreError: 19906:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 541.820484] LustreError: 19906:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 557.535217] Lustre: 3614:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787580101/real 1787580101] req@ffff9913b7bb1500 x1874413081489408/t0(0) o400->MGC192.168.203.104@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787580117 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 557.555218] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 560.105774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 561.701593] LustreError: 19907:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 561.719086] LustreError: 19907:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 569.100309] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 569.108648] Lustre: Skipped 1 previous similar message [ 573.273954] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 574.433417] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 574.443949] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 574.464888] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 583.476483] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 589.792364] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 589.805392] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 589.835069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 593.650548] Lustre: server umount lustre-MDT0000 complete [ 596.960367] LustreError: 21244:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 596.985334] LustreError: 21244:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 597.524382] LustreError: 19887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787580157 with bad export cookie 9166560507614047834 [ 597.538260] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 597.549526] LustreError: 19887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 597.975295] Lustre: server umount lustre-MDT0001 complete [ 607.620869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 612.877260] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 621.828413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 626.935592] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 634.832245] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 641.370337] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 644.081139] Lustre: Failing over lustre-MDT0000 [ 644.369260] Lustre: server umount lustre-MDT0000 complete [ 647.202872] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 647.219241] Lustre: Skipped 2 previous similar messages [ 662.498344] LustreError: 23240:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 662.519338] LustreError: 23240:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 663.348359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 663.519676] Lustre: 3616:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787580207/real 1787580207] req@ffff9913b7bb0700 x1874413081519232/t0(0) o400->MGC192.168.203.104@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787580223 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 663.574202] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 663.590179] Lustre: MGS: Not available for connect from 0@lo (not set up) [ 663.597112] Lustre: Skipped 2 previous similar messages [ 663.877390] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 667.938783] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 667.955790] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 667.968021] Lustre: Skipped 1 previous similar message [ 667.980775] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 668.064817] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 677.400847] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 683.490346] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 683.505159] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 687.892690] Lustre: server umount lustre-MDT0000 complete [ 691.525767] LustreError: 23224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787580251 with bad export cookie 9166560507614048772 [ 691.539045] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 692.034080] Lustre: server umount lustre-MDT0001 complete [ 701.450930] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 10:04:19 (1787580259) [ 718.126457] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 729.005640] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 769.708455] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 781.725804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 781.972508] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 782.000123] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 782.067419] Lustre: lustre-MDT0000: new disk, initializing [ 782.122966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 782.129949] Lustre: Skipped 3 previous similar messages [ 782.139957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 786.526825] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 797.300669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 797.494357] Lustre: 29619:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 797.833061] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 803.005831] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 813.555481] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 820.392410] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 828.204688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 828.452347] Lustre: 31586:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 830.107361] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 830.162387] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 830.222808] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 830.239958] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 835.478052] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 845.717893] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 851.625976] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 876.515297] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 876.520585] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 876.544168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 876.560204] Lustre: Skipped 2 previous similar messages [ 876.569961] Lustre: Skipped 4 previous similar messages [ 880.110339] Lustre: server umount lustre-OST0000 complete [ 884.706099] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 890.383693] Lustre: server umount lustre-MDT0000 complete [ 891.872645] LustreError: 32125:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 891.899710] LustreError: 32125:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 894.152907] LustreError: 29610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787580454 with bad export cookie 9166560507614049178 [ 894.160845] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 894.167976] LustreError: 29610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 894.651712] Lustre: server umount lustre-MDT0001 complete [ 906.942453] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 910.414399] Key type lgssc unregistered [ 910.773604] LNet: 33574:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 910.782840] LNetError: 33574:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 910.813048] LNet: Removed LNI 192.168.203.104@tcp [ 911.917671] Key type .llcrypt unregistered [ 911.928686] Key type ._llcrypt unregistered [ 937.083600] Key type ._llcrypt registered [ 937.086621] Key type .llcrypt registered [ 937.168978] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 938.257036] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 938.295110] alg: No test for adler32 (adler32-zlib) [ 939.319298] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 939.521463] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 941.167195] Key type lgssc registered [ 942.286070] Lustre: Echo OBD driver; http://www.lustre.org/ [ 949.721742] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 949.755244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 951.182060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 955.831118] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 961.521162] LustreError: 34997:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 961.543475] LustreError: 34997:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 965.376932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 965.974270] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 970.694867] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 978.112430] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 983.446348] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 989.988776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 990.345109] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 991.429728] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 996.286215] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 996.568737] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1003.688845] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1016.580976] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 10:09:34 (1787580574) [ 1018.458531] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1020.280162] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1026.553474] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 10:09:44 (1787580584) [ 1034.736643] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 10:09:52 (1787580592) [ 1035.802939] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1044.881204] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1044.887503] Lustre: Skipped 2 previous similar messages [ 1046.870629] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 10:10:04 (1787580604) [ 1049.002601] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1049.008974] Lustre: Skipped 2 previous similar messages [ 1058.480442] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1058.494060] Lustre: Skipped 4 previous similar messages [ 1075.216116] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 10:10:32 (1787580632) [ 1080.380694] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1085.453934] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1094.845951] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 10:10:52 (1787580652) [ 1097.010131] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1097.024863] Lustre: Skipped 6 previous similar messages [ 1109.715920] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 10:11:07 (1787580667) [ 1124.838749] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1124.853271] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1124.866695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1127.666869] Lustre: server umount lustre-OST0000 complete [ 1135.078427] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1135.081094] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1135.091068] Lustre: Skipped 1 previous similar message [ 1135.091780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1135.106094] LustreError: Skipped 1 previous similar message [ 1135.130324] Lustre: Skipped 2 previous similar messages [ 1136.850983] Lustre: server umount lustre-MDT0000 complete [ 1140.196371] LustreError: 36237:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1140.208266] LustreError: 36237:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1140.247369] LustreError: 34978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787580700 with bad export cookie 17275807627208583549 [ 1140.250633] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1140.253718] LustreError: 34978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1140.624401] Lustre: server umount lustre-MDT0001 complete [ 1151.749360] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 1155.580242] Key type lgssc unregistered [ 1155.937288] LNet: 42859:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1155.950645] LNetError: 42859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1155.979038] LNet: Removed LNI 192.168.203.104@tcp [ 1156.990376] Key type .llcrypt unregistered [ 1156.992424] Key type ._llcrypt unregistered [ 1176.734707] Key type ._llcrypt registered [ 1176.736947] Key type .llcrypt registered [ 1176.875915] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 1192.802913] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 1194.064835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1194.103137] alg: No test for adler32 (adler32-zlib) [ 1195.184650] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 1195.504221] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 1197.263152] Key type lgssc registered [ 1198.279863] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1233.399024] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1233.431130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1234.626104] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1234.651586] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1234.728363] Lustre: lustre-MDT0000: new disk, initializing [ 1234.843253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1234.888175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1239.879850] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1250.068717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1250.208499] Lustre: 46012:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1250.286331] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1250.304685] Lustre: Skipped 1 previous similar message [ 1250.439466] Lustre: lustre-MDT0001: new disk, initializing [ 1250.539376] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1250.577988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1250.600858] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1256.057705] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1266.062845] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1271.485480] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1277.920181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1278.332925] Lustre: lustre-OST0000: new disk, initializing [ 1278.340728] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1278.355469] Lustre: 47981:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1278.498342] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1285.841287] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1287.772506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1287.793497] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1288.047555] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1297.857949] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1305.130510] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1305.476295] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1312.452050] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1312.729001] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1318.880511] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1318.883404] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1318.888525] LustreError: Skipped 1 previous similar message [ 1318.911681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1318.917027] Lustre: Skipped 1 previous similar message [ 1318.949872] Lustre: Skipped 1 previous similar message [ 1324.006739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1324.021698] Lustre: Skipped 1 previous similar message [ 1327.022680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1327.033157] Lustre: Skipped 1 previous similar message [ 1329.123068] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1329.301602] Lustre: server umount lustre-OST0000 complete [ 1337.314205] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1337.333775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1337.341634] Lustre: Skipped 1 previous similar message [ 1340.105974] Lustre: server umount lustre-MDT0000 complete [ 1343.907113] LustreError: 47980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787580903 with bad export cookie 1364680793552127973 [ 1343.913586] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1343.922247] LustreError: 47980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1344.382506] Lustre: server umount lustre-MDT0001 complete [ 1354.144343] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 10:15:12 (1787580912) [ 1361.213363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1361.728366] LustreError: 50446:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1361.745919] LustreError: 50446:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1361.801495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1367.015048] LustreError: 50447:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1367.042465] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1372.134937] LustreError: 50446:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1375.713313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1376.121405] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1380.555711] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1388.354782] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1393.998647] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1400.794746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1401.250362] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1408.305280] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1416.365052] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1424.292439] LustreError: 52976:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1425.275872] Lustre: Modifying parameter general.timeout=1 in log params [ 1426.390972] Lustre: Modifying parameter general.timeout=2 in log params [ 1429.245102] Lustre: Modifying parameter general.timeout=5 in log params [ 1429.257477] Lustre: Skipped 2 previous similar messages [ 1433.704814] Lustre: Modifying parameter general.timeout=10 in log params [ 1433.707590] Lustre: Skipped 4 previous similar messages [ 1442.039950] Lustre: Modifying parameter general.timeout=19 in log params [ 1442.047885] Lustre: Skipped 8 previous similar messages [ 1458.430964] Lustre: Modifying parameter general.timeout=39 in log params [ 1458.437085] Lustre: Skipped 19 previous similar messages [ 1490.641116] Lustre: Modifying parameter general.timeout=71 in log params [ 1490.644042] Lustre: Skipped 31 previous similar messages [ 1561.148186] Lustre: Disabling parameter general.timeout= in log params [ 1561.151649] Lustre: Skipped 58 previous similar messages [ 1564.967699] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 10:18:42 (1787581122) [ 1567.878798] Lustre: 59928:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1569.696715] Lustre: 60035:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1573.427523] Lustre: 60235:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1573.441912] Lustre: 60235:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1607.019093] Lustre: 61804:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1607.028704] Lustre: 61804:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1607.058429] LustreError: 61804:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1613.004703] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 10:19:30 (1787581170) [ 1630.182350] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1630.183491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1630.197592] Lustre: Skipped 3 previous similar messages [ 1630.201783] Lustre: Skipped 1 previous similar message [ 1634.242118] Lustre: server umount lustre-MDT0000 complete [ 1635.305470] LustreError: 50447:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1635.327122] LustreError: 50447:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1638.029170] LustreError: 50427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787581197 with bad export cookie 1364680793552128848 [ 1638.031709] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1638.045712] LustreError: 50427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1638.516900] Lustre: server umount lustre-MDT0001 complete [ 1653.050499] Lustre: server umount lustre-OST0000 complete [ 1681.718179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1681.991233] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1682.239289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1686.833989] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1695.707274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1695.924927] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1696.008745] Lustre: 64341:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1700.667711] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1708.207966] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1714.230795] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1720.670019] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1720.982189] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1721.312926] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1721.317901] Lustre: Skipped 1 previous similar message [ 1727.379658] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1734.074910] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1735.763396] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1735.772968] Lustre: Skipped 23 previous similar messages [ 1735.839611] Lustre: 66908:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1742.832653] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1742.848851] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1742.858344] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1748.005802] Lustre: server umount lustre-OST0000 complete [ 1752.545911] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1752.558385] LustreError: Skipped 1 previous similar message [ 1752.570072] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1752.605119] Lustre: Skipped 1 previous similar message [ 1763.811194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1763.825680] Lustre: Skipped 8 previous similar messages [ 1766.367687] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1766.763400] Lustre: server umount lustre-MDT0000 complete [ 1768.931624] LustreError: 64353:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1768.961881] LustreError: 64353:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1771.360560] LustreError: 65081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787581331 with bad export cookie 1364680793552132040 [ 1771.366627] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1771.392366] LustreError: 65081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1771.942359] Lustre: server umount lustre-MDT0001 complete [ 1787.785859] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 1791.519205] Key type lgssc unregistered [ 1791.969581] LNet: 68121:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1791.986266] LNetError: 68121:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1792.012113] LNet: Removed LNI 192.168.203.104@tcp [ 1793.041699] Key type .llcrypt unregistered [ 1793.045500] Key type ._llcrypt unregistered [ 1822.846201] Key type ._llcrypt registered [ 1822.849556] Key type .llcrypt registered [ 1822.962182] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 1824.152355] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1824.223150] alg: No test for adler32 (adler32-zlib) [ 1825.257346] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 1825.540506] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 1827.215674] Key type lgssc registered [ 1828.680943] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1837.122992] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1837.178290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1838.779226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1843.320839] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1848.311822] LustreError: 69538:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1848.343549] LustreError: 69538:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1852.528663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1853.074481] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1858.004255] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1867.182858] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1873.194306] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1880.605791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1881.304117] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1887.501228] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1895.618130] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1904.672880] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1918.434101] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1918.453962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1918.488707] Lustre: Skipped 2 previous similar messages [ 1918.518555] Lustre: Skipped 2 previous similar messages [ 1922.518977] Lustre: server umount lustre-MDT0000 complete [ 1923.554854] LustreError: 69537:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1923.586662] LustreError: 69537:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1927.396161] LustreError: 69518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787581487 with bad export cookie 5421564251799949490 [ 1927.406582] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1927.418200] LustreError: 69518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1927.858550] Lustre: server umount lustre-MDT0001 complete [ 1943.206276] Lustre: server umount lustre-OST0000 complete [ 1972.905148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1973.391910] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1973.441636] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1973.840985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1978.920572] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1989.520757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1989.706334] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1989.744996] Lustre: 74413:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1990.175729] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1995.300926] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2004.113404] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2010.546600] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2018.200095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2018.555683] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2019.087870] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2026.515613] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2034.731510] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2037.208746] Lustre: 76984:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2040.033952] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2048.997553] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2049.028612] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2049.081901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2049.543995] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2053.022792] Lustre: server umount lustre-OST0000 complete [ 2059.745670] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2059.766694] Lustre: Skipped 1 previous similar message [ 2059.780871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2059.789659] Lustre: Skipped 3 previous similar messages [ 2064.013972] Lustre: server umount lustre-MDT0000 complete [ 2064.866322] LustreError: 74419:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2064.898972] LustreError: 74419:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2068.542052] LustreError: 74403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787581628 with bad export cookie 5421564251799950337 [ 2068.547967] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2068.556074] LustreError: 74403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2069.091872] Lustre: server umount lustre-MDT0001 complete [ 2082.218492] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 2085.827851] Key type lgssc unregistered [ 2086.212372] LNet: 78201:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2086.228673] LNetError: 78201:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2086.268309] LNet: Removed LNI 192.168.203.104@tcp [ 2087.573084] Key type .llcrypt unregistered [ 2087.578619] Key type ._llcrypt unregistered [ 2117.424612] Key type ._llcrypt registered [ 2117.429607] Key type .llcrypt registered [ 2117.593300] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 2119.283615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2119.407026] alg: No test for adler32 (adler32-zlib) [ 2120.541538] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 2121.008685] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 2122.778466] Key type lgssc registered [ 2124.476509] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2134.248079] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2134.321538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2136.241118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2141.883423] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2145.763589] LustreError: 79614:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2145.804638] LustreError: 79614:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2150.889724] LustreError: 79615:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2152.064696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2152.846438] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2157.934103] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2165.689148] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2170.672379] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2176.784837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2177.370854] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2183.963726] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2191.114054] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2197.634675] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2198.556562] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2200.285546] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2200.292726] Lustre: Skipped 1 previous similar message [ 2203.220890] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2203.223799] Lustre: Skipped 2 previous similar messages [ 2207.607847] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2207.612814] Lustre: Skipped 3 previous similar messages [ 2216.378225] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2216.399358] Lustre: Skipped 7 previous similar messages [ 2233.140815] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2233.145282] Lustre: Skipped 12 previous similar messages [ 2265.650661] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2265.658878] Lustre: Skipped 31 previous similar messages [ 2311.695812] Lustre: 87728:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2314.438617] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 10:31:12 (1787581872) [ 2315.913018] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2317.939289] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 10:31:15 (1787581875) [ 2323.937336] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2323.950358] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2323.954537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2323.961486] Lustre: Skipped 1 previous similar message [ 2323.972576] Lustre: Skipped 1 previous similar message [ 2328.637831] Lustre: server umount lustre-OST0000 complete [ 2334.185610] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2334.187467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2334.197504] Lustre: Skipped 1 previous similar message [ 2334.211171] Lustre: Skipped 1 previous similar message [ 2338.748670] Lustre: server umount lustre-MDT0000 complete [ 2339.307613] LustreError: 79609:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2339.349423] LustreError: 79609:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2342.337346] LustreError: 79596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787581902 with bad export cookie 5573146322647922960 [ 2342.341368] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2342.349421] LustreError: 79596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2342.665961] Lustre: server umount lustre-MDT0001 complete [ 2353.756539] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 2356.331397] Key type lgssc unregistered [ 2356.669503] LNet: 88979:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2356.677322] LNetError: 88979:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2356.696701] LNet: Removed LNI 192.168.203.104@tcp [ 2357.471618] Key type .llcrypt unregistered [ 2357.473992] Key type ._llcrypt unregistered [ 2361.805166] Key type ._llcrypt registered [ 2361.807571] Key type .llcrypt registered [ 2361.963371] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2365.759716] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules [ 2366.568484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2366.599398] alg: No test for adler32 (adler32-zlib) [ 2367.622514] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 2367.836414] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 2369.503156] Key type lgssc registered [ 2370.782940] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2371.760205] LustreError: 90181:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2389.493716] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 2394.063309] LustreError: 90181:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2400.008439] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2400.022873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2401.516705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2405.793462] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2412.000350] LustreError: 91205:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2412.029253] LustreError: 91205:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2414.810625] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 10:32:52 (1787581972) [ 2417.128298] LustreError: 91206:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2419.141820] LustreError: 91237:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2419.154391] LustreError: 91237:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2419.163083] LustreError: 91237:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 2419.409383] Lustre: server umount lustre-MDT0000 complete [ 2432.809810] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 2435.979167] Key type lgssc unregistered [ 2436.402085] LNet: 92734:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2436.425624] LNetError: 92734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2436.443668] LNet: Removed LNI 192.168.203.104@tcp [ 2437.286993] Key type .llcrypt unregistered [ 2437.295604] Key type ._llcrypt unregistered [ 2459.067076] Key type ._llcrypt registered [ 2459.070029] Key type .llcrypt registered [ 2459.201608] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 2460.847801] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2460.903224] alg: No test for adler32 (adler32-zlib) [ 2462.118485] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 2462.504147] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 2464.368702] Key type lgssc registered [ 2465.662196] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2474.500746] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2474.530174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2476.348674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2481.313642] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2486.759233] LustreError: 93954:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2486.792730] LustreError: 93954:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2490.429095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2491.097805] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2495.885730] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2504.020224] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2510.891756] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2518.165074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2518.999123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2525.594455] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2533.785175] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2584.879235] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 10:35:42 (1787582142) [ 2586.698973] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2588.567254] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 10:35:46 (1787582146) [ 2593.251763] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2593.256327] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2593.280482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2596.839576] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2596.840213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2596.863510] Lustre: Skipped 2 previous similar messages [ 2599.492491] Lustre: server umount lustre-MDT0000 complete [ 2601.952471] LustreError: 96747:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2601.973124] LustreError: 96747:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2603.173925] LustreError: 93935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787582163 with bad export cookie 12200613591030165694 [ 2603.186274] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2603.693824] Lustre: server umount lustre-MDT0001 complete [ 2619.053193] Lustre: server umount lustre-OST0000 complete [ 2630.398150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2630.752205] LustreError: 98254:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2630.931426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2635.588876] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2636.258530] LustreError: 98255:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2640.361260] LustreError: 98254:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2643.977650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2644.800410] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2649.570859] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2657.894687] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2664.342093] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2677.543534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2677.734614] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2677.748516] LustreError: 98241:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2677.783248] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2677.804888] LustreError: 100336:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2677.812879] LustreError: 100336:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2677.819715] LustreError: 100336:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2677.922850] Lustre: server umount lustre-OST0000 complete [ 2677.925973] LustreError: 100336:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2687.172397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2687.361523] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2687.370686] LustreError: 98241:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2687.408388] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2687.416623] LustreError: 100803:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2687.422958] LustreError: 100803:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2687.428618] LustreError: 100803:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2687.507381] Lustre: server umount lustre-OST0000 complete [ 2687.511214] LustreError: 100803:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2697.026449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2697.302962] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2697.309722] Lustre: Found index 0 for lustre-OST0000, updating log [ 2697.321208] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2697.403741] Lustre: lustre-OST0000: new disk, initializing [ 2697.415756] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2697.426445] Lustre: 101338:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2697.790640] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2704.154561] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2708.402217] Lustre: Failing over lustre-OST0000 [ 2708.872300] Lustre: server umount lustre-OST0000 complete [ 2715.111593] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2715.130966] Lustre: Skipped 1 previous similar message [ 2715.140612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2715.157115] Lustre: Skipped 1 previous similar message [ 2720.225579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2720.241658] Lustre: Skipped 1 previous similar message [ 2725.344038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2725.350861] Lustre: Skipped 1 previous similar message [ 2727.391227] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2727.582196] Lustre: server umount lustre-MDT0000 complete [ 2730.477808] LustreError: 98251:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2730.508359] LustreError: 98251:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2731.892068] LustreError: 98236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787582291 with bad export cookie 12200613591030168431 [ 2731.897860] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2731.911476] LustreError: 98236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2732.336199] Lustre: server umount lustre-MDT0001 complete [ 2743.208112] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 10:38:21 (1787582301) [ 2764.442843] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 2776.245662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2776.680618] LustreError: 103684:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2776.763824] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2781.134254] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2790.387390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2795.803892] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2798.873380] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2805.830192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2806.574378] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2813.618590] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2815.783613] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2815.796915] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2815.826335] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2821.104663] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 2821.132256] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2822.197775] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:53 [ 2822.240686] Lustre: lustre-OST0000: Recovery over after 0:07, of 1 clients 1 recovered and 0 were evicted. [ 2822.249674] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2822.275762] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2823.732854] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2823.838118] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2823.925779] Lustre: lustre-OST0001: new disk, initializing [ 2823.929519] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2823.944882] Lustre: 106003:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2824.279154] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2824.294592] Lustre: Skipped 2 previous similar messages [ 2830.427825] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2830.440321] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2830.546671] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2831.085649] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2839.259717] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2844.391937] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2849.703844] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.4@tcp (stopping) [ 2851.302068] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2851.344675] Lustre: Skipped 3 previous similar messages [ 2863.072869] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2863.362976] Lustre: server umount lustre-MDT0001 complete [ 2864.956339] LustreError: 106317:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2864.972070] LustreError: 106317:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2876.513237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2876.672817] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2876.687301] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2876.758810] Lustre: 103673:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2876.898105] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2876.900890] Lustre: Skipped 14 previous similar messages [ 2877.995970] LustreError: 107526:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2878.021600] LustreError: 107526:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2878.041173] LustreError: 103676:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2878.253192] LustreError: 107541:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.104@tcp: cfg command failed: rc = -17 [ 2878.273988] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2878.273988] [ 2883.063488] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2883.350771] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2894.176469] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 10:40:51 (1787582451) [ 2911.962105] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 2931.549564] Lustre: 109625:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2960.689070] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2965.087298] Lustre: 110761:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3056.610660] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3056.616143] LustreError: Skipped 1 previous similar message [ 3056.619714] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3056.628972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3062.526520] Lustre: server umount lustre-MDT0000 complete [ 3066.928735] LustreError: 103666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787582626 with bad export cookie 12200613591030169439 [ 3066.945473] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3066.958492] LustreError: 103666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3067.363617] LustreError: 110942:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3067.369609] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3067.382642] LustreError: 110942:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 3067.412857] Lustre: Skipped 3 previous similar messages [ 3067.455900] Lustre: server umount lustre-MDT0001 complete [ 3074.607897] Lustre: server umount lustre-OST0000 complete [ 3078.936357] Lustre: server umount lustre-OST0001 complete [ 3085.093522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3095.856341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3104.672893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3114.812631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3139.592907] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 3151.960546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3151.991515] Lustre: lustre-MDT0000: reset Object Index mappings [ 3152.682285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3152.686675] Lustre: Skipped 1 previous similar message [ 3157.432487] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3167.247151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3167.267987] Lustre: lustre-MDT0001: reset Object Index mappings [ 3174.155698] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3178.130674] Lustre: 115866:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3186.771445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3192.831539] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:195 to 0x280000400:225) [ 3197.017914] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3197.936828] LustreError: 116223:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3197.963098] LustreError: 116223:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 3197.969148] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:386 to 0x280000401:418) [ 3207.360596] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3213.297088] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3213.305600] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3215.890453] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3224.698774] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3229.260359] Lustre: 117744:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3283.239241] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 10:47:21 (1787582841) [ 3308.523939] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3308.526635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3308.566658] Lustre: Skipped 3 previous similar messages [ 3308.579047] Lustre: Skipped 11 previous similar messages [ 3322.335120] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3324.568177] Lustre: server umount lustre-MDT0000 complete [ 3328.431656] LustreError: 114701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787582888 with bad export cookie 12200613591030216507 [ 3328.460035] LustreError: 114701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3328.463225] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3335.138400] Lustre: server umount lustre-MDT0001 complete [ 3345.985066] Lustre: server umount lustre-OST0000 complete [ 3364.319179] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3364.530990] Lustre: server umount lustre-OST0001 complete [ 3372.311131] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 3384.339462] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 3426.149627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3426.402947] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3426.448127] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3426.540107] Lustre: lustre-MDT0000: new disk, initializing [ 3426.615319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3426.625921] Lustre: Skipped 3 previous similar messages [ 3426.635986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3431.494827] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3439.270135] Lustre: server umount lustre-MDT0000 complete [ 3449.775217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3450.102319] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3450.128449] Lustre: 122018:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3450.146092] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3455.918985] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3461.110757] Lustre: server umount lustre-MDT0000 complete [ 3470.634955] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 10:50:28 (1787583028) [ 3472.098422] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3474.089360] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 10:50:31 (1787583031) [ 3491.890136] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 3502.527589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3508.090594] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3516.820639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3516.973616] Lustre: 124067:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3517.028881] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3517.031947] Lustre: Skipped 1 previous similar message [ 3517.141717] Lustre: lustre-MDT0001: new disk, initializing [ 3517.276047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3517.290074] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3522.020711] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3526.892969] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3533.470275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3533.824105] Lustre: lustre-OST0000: new disk, initializing [ 3533.830281] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3533.848658] Lustre: 125644:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3534.929172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3534.936167] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3535.053037] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3540.531428] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3550.348682] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3550.571246] Lustre: lustre-OST0001: new disk, initializing [ 3550.575473] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3550.583009] Lustre: 126515:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3552.237374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3552.248428] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3552.380477] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3557.410874] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3567.230585] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3571.118091] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3980.676455] Lustre: 132937:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3980.683591] Lustre: 132937:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3980.688805] Lustre: 132937:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3980.693838] Lustre: 132937:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3980.698714] Lustre: 132937:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3980.704488] Lustre: 132937:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4285.497930] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 11:04:02 (1787583842) [ 4301.283993] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4301.297673] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4301.331912] Lustre: Skipped 3 previous similar messages [ 4301.338399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4301.350421] Lustre: Skipped 12 previous similar messages [ 4307.590328] Lustre: server umount lustre-MDT0000 complete [ 4307.942371] LustreError: 127338:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4307.962803] LustreError: 127338:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 4311.379930] LustreError: 124060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787583871 with bad export cookie 12200613591030284939 [ 4311.395191] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4311.413333] LustreError: 124060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4313.058085] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4313.084677] Lustre: Skipped 4 previous similar messages [ 4318.043295] Lustre: server umount lustre-MDT0001 complete [ 4337.119126] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4337.333991] Lustre: server umount lustre-OST0000 complete [ 4347.471928] Lustre: server umount lustre-OST0001 complete [ 4355.569802] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 4367.552502] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 4410.762647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4411.021537] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4411.055649] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4411.140660] Lustre: lustre-MDT0000: new disk, initializing [ 4411.219534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4411.229061] Lustre: Skipped 5 previous similar messages [ 4411.248298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4415.605806] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4427.503030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4427.680239] Lustre: 136011:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4427.757685] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4427.763688] Lustre: Skipped 1 previous similar message [ 4427.863596] Lustre: lustre-MDT0001: new disk, initializing [ 4427.987305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4428.003613] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4432.780105] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4444.114409] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4450.834636] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4458.219873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4458.581244] Lustre: lustre-OST0000: new disk, initializing [ 4458.585551] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4458.593796] Lustre: 137979:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4458.699926] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4458.708237] Lustre: Skipped 1 previous similar message [ 4459.877462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4459.892294] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4460.082041] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4465.374949] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4476.329502] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4486.998826] Lustre: lustre-MDD0000: changelog on [ 4490.163523] Lustre: lustre-MDD0001: changelog on [ 6559.362314] Lustre: lustre-MDD0001: changelog off [ 6561.487071] Lustre: 137819:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6562.506797] Lustre: lustre-MDD0000: changelog off [ 6569.441845] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6569.454852] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6569.469047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6569.473040] Lustre: Skipped 9 previous similar messages [ 6572.520962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6572.528757] Lustre: Skipped 1 previous similar message [ 6574.602412] Lustre: server umount lustre-OST0000 complete [ 6578.659531] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6578.669524] LustreError: Skipped 1 previous similar message [ 6578.673557] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6578.691622] Lustre: Skipped 1 previous similar message [ 6578.695451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6582.754057] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6584.673666] Lustre: server umount lustre-MDT0000 complete [ 6587.880129] LustreError: 136021:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6587.902325] LustreError: 136021:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6588.042776] LustreError: 136003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787586147 with bad export cookie 12200613591031105073 [ 6588.047191] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6588.054242] LustreError: 136003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6588.725387] Lustre: server umount lustre-MDT0001 complete [ 6603.039710] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 6606.898533] Key type lgssc unregistered [ 6607.213486] LNet: 147152:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6607.232029] LNetError: 147152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6607.265889] LNet: Removed LNI 192.168.203.104@tcp [ 6608.700161] Key type .llcrypt unregistered [ 6608.705541] Key type ._llcrypt unregistered [ 6633.449558] Key type ._llcrypt registered [ 6633.454196] Key type .llcrypt registered [ 6633.711917] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 6650.574956] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 6651.933949] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6652.046300] alg: No test for adler32 (adler32-zlib) [ 6653.239660] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 6653.529538] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 6655.295306] Key type lgssc registered [ 6656.656782] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6693.590233] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6693.630721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6694.860464] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6694.897779] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6694.958992] Lustre: lustre-MDT0000: new disk, initializing [ 6695.039849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6695.068677] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6699.797230] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6710.336888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6710.491584] Lustre: 150355:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6710.575814] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6710.589650] Lustre: Skipped 1 previous similar message [ 6710.722818] Lustre: lustre-MDT0001: new disk, initializing [ 6710.828905] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6710.861869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6710.876219] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6715.543437] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6726.254975] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6732.638499] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6740.431676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6740.762463] Lustre: lustre-OST0000: new disk, initializing [ 6740.767658] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6740.776877] Lustre: 152323:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6740.888765] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6747.591563] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6748.262039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6748.286868] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6748.469310] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6758.222331] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6764.180438] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_module obdecho/obdecho [ 6766.226593] LustreError: 153328:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6766.238901] LustreError: 153328:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6768.367074] LustreError: 153424:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6782.439873] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6782.450380] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6782.485589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6784.484782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6784.499078] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6784.517563] Lustre: Skipped 1 previous similar message [ 6787.595114] Lustre: server umount lustre-MDT0000 complete [ 6792.457733] LustreError: 150348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787586352 with bad export cookie 6242377648450883680 [ 6792.484154] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6792.934871] Lustre: server umount lustre-MDT0001 complete [ 6809.306706] Lustre: server umount lustre-OST0000 complete [ 6835.798725] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 6847.832844] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 6886.077761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6886.349177] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6886.365494] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6886.421757] Lustre: lustre-MDT0000: new disk, initializing [ 6886.473821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6886.485646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6891.239207] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6902.659845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6902.868078] Lustre: 157143:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6902.938995] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6902.941951] Lustre: Skipped 1 previous similar message [ 6903.051649] Lustre: lustre-MDT0001: new disk, initializing [ 6903.201402] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6903.267961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6903.288551] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6907.381719] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6919.726851] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6925.862415] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6933.141916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6933.615420] Lustre: lustre-OST0000: new disk, initializing [ 6933.622588] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6933.631996] Lustre: 159113:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6933.758549] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6939.795711] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6940.726843] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6940.731797] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6940.825098] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6949.489653] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6954.408179] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6954.622271] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6959.142867] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6959.289221] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6961.632303] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6961.638640] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6961.647174] LustreError: Skipped 1 previous similar message [ 6961.648461] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6961.648466] Lustre: Skipped 2 previous similar messages [ 6961.688345] Lustre: Skipped 1 previous similar message [ 6966.752660] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6966.773063] Lustre: Skipped 2 previous similar messages [ 6968.051662] Lustre: server umount lustre-OST0000 complete [ 6974.944504] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6974.950786] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6974.960589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6981.191871] Lustre: server umount lustre-MDT0000 complete [ 6982.113146] LustreError: 157156:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6982.148402] LustreError: 157156:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6985.670555] LustreError: 157940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787586545 with bad export cookie 6242377648450884639 [ 6985.673760] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6985.682309] LustreError: 157940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 6985.983894] Lustre: server umount lustre-MDT0001 complete [ 6990.520396] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 11:49:08 (1787586548) [ 6993.034450] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6995.303607] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 11:49:12 (1787586552) [ 7005.140947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7005.754530] LustreError: 161526:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7005.823559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7011.301729] LustreError: 161527:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7011.399824] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7015.409819] LustreError: 161526:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7020.514725] LustreError: 161527:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7021.044131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7026.406608] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7034.145468] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7040.006837] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7047.677805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7048.241325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7048.252807] Lustre: Skipped 1 previous similar message [ 7055.379897] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7063.626490] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7076.836178] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7076.841925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7076.861835] Lustre: Skipped 2 previous similar messages [ 7076.870641] Lustre: Skipped 4 previous similar messages [ 7082.969334] Lustre: server umount lustre-MDT0000 complete [ 7084.930779] LustreError: 162699:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7087.804058] LustreError: 161508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787586647 with bad export cookie 6242377648450885521 [ 7087.811613] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7087.821345] LustreError: 161508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7092.192434] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7092.218820] Lustre: Skipped 1 previous similar message [ 7094.610519] Lustre: server umount lustre-MDT0001 complete [ 7102.636969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7119.361701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7119.722602] LustreError: 165389:0:(llog.c:1655:llog_backup()) MGC192.168.203.104@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7119.736527] Lustre: 165389:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.104@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7119.870610] LustreError: 165394:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7119.905040] LustreError: 165394:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7153.633088] LustreError: 165394:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7153.665293] LustreError: 165394:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 24 previous similar messages [ 7180.255304] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7180.257830] Lustre: 165389:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.104@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7218.657398] LustreError: 165424:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7218.668919] LustreError: 165424:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 42 previous similar messages [ 7221.240845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7226.340594] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7226.843912] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7239.167352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7248.874617] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7259.105373] LustreError: 148577:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b700d2e5 x1874420001022336/t0(0) o250->MGC192.168.203.104@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7264.760954] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7265.408328] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7274.114290] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7276.154590] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7283.217477] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7285.277355] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7310.823747] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7310.824775] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7310.851476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7310.851489] Lustre: Skipped 6 previous similar messages [ 7310.866600] Lustre: Skipped 2 previous similar messages [ 7314.209472] Lustre: server umount lustre-MDT0000 complete [ 7318.379433] LustreError: 161507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787586878 with bad export cookie 6242377648450892010 [ 7318.381055] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7318.394731] LustreError: 161507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7319.224184] Lustre: server umount lustre-MDT0001 complete [ 7334.234574] Lustre: server umount lustre-OST0000 complete [ 7362.417780] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 7374.601282] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 7420.861350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7421.233817] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7421.284355] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7421.442263] Lustre: lustre-MDT0000: new disk, initializing [ 7421.576273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7421.581889] Lustre: Skipped 1 previous similar message [ 7421.606126] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7427.203697] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7439.884626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7440.044528] Lustre: 170504:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7440.473426] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7445.731906] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7455.963831] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7463.728823] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7471.359120] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7471.813779] Lustre: lustre-OST0000: new disk, initializing [ 7471.817969] Lustre: Skipped 1 previous similar message [ 7471.823662] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7471.829702] Lustre: Skipped 2 previous similar messages [ 7471.842519] Lustre: 172472:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7477.313497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7477.324616] Lustre: Skipped 1 previous similar message [ 7477.334398] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7477.458353] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7479.596508] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7490.540436] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7495.991827] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7496.340740] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7501.338892] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7501.593234] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7508.450466] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7508.453529] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7508.463102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7508.463111] Lustre: Skipped 2 previous similar messages [ 7508.465754] LustreError: Skipped 1 previous similar message [ 7508.504152] Lustre: Skipped 1 previous similar message [ 7510.102286] Lustre: server umount lustre-OST0000 complete [ 7520.987967] Lustre: server umount lustre-MDT0000 complete [ 7523.814380] LustreError: 170515:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7523.835232] LustreError: 170515:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 39 previous similar messages [ 7525.332575] LustreError: 170496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787587085 with bad export cookie 6242377648450908817 [ 7525.343402] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7525.357919] LustreError: 170496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7526.088573] Lustre: server umount lustre-MDT0001 complete [ 7537.483591] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 11:58:15 (1787587095) [ 7545.370609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7550.568628] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7560.147358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7565.304411] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7575.308222] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7584.220404] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7592.171247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7600.558836] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7609.711935] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7636.588903] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 11:59:53 (1787587193) [ 7647.212234] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7647.222321] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7647.229813] LustreError: Skipped 2 previous similar messages [ 7647.235448] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7647.284600] Lustre: Skipped 3 previous similar messages [ 7647.345916] Lustre: Skipped 5 previous similar messages [ 7651.008840] Lustre: server umount lustre-OST0000 complete [ 7663.747600] Lustre: server umount lustre-MDT0000 complete [ 7668.564983] LustreError: 177600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787587228 with bad export cookie 6242377648450909692 [ 7668.583430] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7668.586321] LustreError: 177600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7669.030861] Lustre: server umount lustre-MDT0001 complete [ 7682.244878] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 7686.218737] Key type lgssc unregistered [ 7686.607776] LNet: 178646:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7686.621584] LNetError: 178646:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7686.645522] LNet: Removed LNI 192.168.203.104@tcp [ 7687.975559] Key type .llcrypt unregistered [ 7687.978272] Key type ._llcrypt unregistered [ 7713.668552] Key type ._llcrypt registered [ 7713.671501] Key type .llcrypt registered [ 7713.782657] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 7714.804511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7715.040162] alg: No test for adler32 (adler32-zlib) [ 7716.305740] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 7716.737568] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 7718.487192] Key type lgssc registered [ 7720.162732] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7729.599505] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7729.667797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7745.375957] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7750.495923] LustreError: 179902:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.104@tcp: failed processing log, type 4: rc = -110 [ 7755.617206] LustreError: 179902:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7755.630113] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7755.636697] LustreError: MGC192.168.203.104@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7755.653198] LustreError: 179902:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7755.665735] LustreError: 179902:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -22 [ 7755.676434] LustreError: 179902:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 7755.757725] Lustre: server umount lustre-OST0000 complete [ 7755.762043] LustreError: 179902:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7766.842686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7768.687145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7774.599664] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7778.284043] LustreError: 180402:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7778.331394] LustreError: 180402:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7783.393909] LustreError: 180403:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7785.846325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7786.403675] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7792.213320] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7801.191182] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7807.938630] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7815.213560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7815.484157] LustreError: 182219:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7815.497249] Lustre: 182219:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.203.104@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7815.639378] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7821.915533] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7828.973563] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7832.419108] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7832.425612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7832.457982] Lustre: Skipped 2 previous similar messages [ 7832.487891] Lustre: Skipped 2 previous similar messages [ 7837.670710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7837.677747] Lustre: Skipped 1 previous similar message [ 7837.882927] Lustre: server umount lustre-MDT0000 complete [ 7841.785017] LustreError: 180381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787587401 with bad export cookie 14993926654998984051 [ 7841.795379] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7841.799389] LustreError: 180381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7842.719347] LustreError: 181132:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000568f7186 x1874421115547904/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7842.795233] LustreError: 180398:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7842.799886] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7843.009856] Lustre: server umount lustre-MDT0001 complete [ 7849.544882] Lustre: server umount lustre-OST0000 complete [ 7859.689944] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 7870.330810] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 7908.375312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7908.704367] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7908.767355] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7908.856768] Lustre: lustre-MDT0000: new disk, initializing [ 7909.051158] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7909.098233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7914.092984] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7926.176784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7926.392345] Lustre: 185741:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7926.512317] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7926.515912] Lustre: Skipped 1 previous similar message [ 7926.683890] Lustre: lustre-MDT0001: new disk, initializing [ 7926.851821] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7926.899584] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7926.917176] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7932.591928] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7944.279181] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7951.555764] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7960.259064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7960.814702] Lustre: lustre-OST0000: new disk, initializing [ 7960.830475] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7960.840662] Lustre: 187713:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7960.948505] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7968.358660] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7968.828802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7968.843190] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7969.043987] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7979.251332] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7985.997656] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7986.339336] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7992.519050] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7992.789969] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7999.971933] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7999.978658] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7999.994106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7999.996908] Lustre: Skipped 1 previous similar message [ 8000.029889] Lustre: Skipped 2 previous similar messages [ 8001.874509] Lustre: server umount lustre-OST0000 complete [ 8008.672388] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8008.686779] LustreError: Skipped 1 previous similar message [ 8008.692949] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8008.708835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8013.135809] Lustre: server umount lustre-MDT0000 complete [ 8015.335522] LustreError: 185754:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8015.376167] LustreError: 185754:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8017.229714] LustreError: 185734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787587577 with bad export cookie 14993926654998984765 [ 8017.238259] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8017.249852] LustreError: 185734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8017.640906] Lustre: server umount lustre-MDT0001 complete [ 8029.460976] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 12:06:27 (1787587587) [ 8048.857215] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 8052.161715] Key type lgssc unregistered [ 8052.460518] LNet: 190664:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8052.468568] LNetError: 190664:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8052.493494] LNet: Removed LNI 192.168.203.104@tcp [ 8053.449956] Key type .llcrypt unregistered [ 8053.456249] Key type ._llcrypt unregistered [ 8071.512249] Key type ._llcrypt registered [ 8071.516407] Key type .llcrypt registered [ 8071.631672] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 8072.708772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8072.815317] alg: No test for adler32 (adler32-zlib) [ 8073.895207] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 8074.145606] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 8075.855281] Key type lgssc registered [ 8077.162337] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8085.268083] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8085.290829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8086.889406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8091.772832] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 8096.244662] LustreError: 191881:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8096.270630] LustreError: 191881:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8100.521927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8100.885095] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8105.647251] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 8114.062948] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8119.994487] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8128.403834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8128.636808] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8128.639471] LustreError: 193746:0:(llog.c:1696:llog_backup()) MGC192.168.203.104@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8128.647290] Lustre: 193746:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.203.104@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8128.796225] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8135.691462] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 8145.086509] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8150.504666] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8150.511317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8150.545916] Lustre: Skipped 2 previous similar messages [ 8150.575233] Lustre: Skipped 2 previous similar messages [ 8154.016437] Lustre: server umount lustre-MDT0000 complete [ 8155.621601] LustreError: 191881:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8155.656574] LustreError: 191881:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8158.122590] LustreError: 191862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787587717 with bad export cookie 5890976850335022165 [ 8158.134744] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8158.135211] LustreError: 191862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8158.558624] Lustre: server umount lustre-MDT0001 complete [ 8173.330475] Lustre: server umount lustre-OST0000 complete [ 8186.669189] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 8201.525926] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 8244.113720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8244.326586] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8244.343937] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8244.406537] Lustre: lustre-MDT0000: new disk, initializing [ 8244.452492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8244.470596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8250.391360] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 8264.440149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8264.553033] Lustre: 197291:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8264.636642] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8264.644087] Lustre: Skipped 1 previous similar message [ 8264.647164] LustreError: 198078:0:(llog.c:1696:llog_backup()) MGC192.168.203.104@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8264.657869] LustreError: 198078:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8264.662515] Lustre: 198078:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.104@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8264.676554] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8264.680121] Lustre: Skipped 1 previous similar message [ 8264.751608] Lustre: lustre-MDT0001: new disk, initializing [ 8264.851769] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8264.892363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8264.902893] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8271.309598] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 8281.599576] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8287.995818] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8295.717036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8295.961065] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8295.964117] Lustre: Skipped 1 previous similar message [ 8295.967512] LustreError: 199241:0:(llog.c:1696:llog_backup()) MGC192.168.203.104@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8295.972654] LustreError: 199241:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8295.976750] Lustre: 199241:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.104@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8295.983332] Lustre: 199241:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8295.999855] Lustre: lustre-OST0000: new disk, initializing [ 8296.003644] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8296.014506] Lustre: 199264:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8296.107249] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8302.144234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8302.166192] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8302.295291] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8302.448677] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 8312.468949] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8317.461994] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8317.677221] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8322.156681] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8322.425272] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8328.166253] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8328.166308] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8328.170301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8328.205705] Lustre: Skipped 1 previous similar message [ 8330.495143] Lustre: server umount lustre-OST0000 complete [ 8336.864338] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8336.870032] LustreError: Skipped 1 previous similar message [ 8336.874270] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8336.891946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8336.912569] Lustre: Skipped 1 previous similar message [ 8341.062232] Lustre: server umount lustre-MDT0000 complete [ 8343.520855] LustreError: 198440:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8343.537403] LustreError: 198440:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8344.873502] LustreError: 197283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787587904 with bad export cookie 5890976850335022886 [ 8344.883864] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8344.889287] LustreError: 197283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8345.369061] Lustre: server umount lustre-MDT0001 complete [ 8355.486840] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 12:11:53 (1787587913) [ 8374.677106] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 8387.178476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8387.661241] LustreError: 202393:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8387.799333] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8393.198283] LustreError: 202394:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8394.899653] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 8402.405892] LustreError: 202393:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8402.422147] LustreError: 202393:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8402.984883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8407.757230] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 8411.078542] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8417.533083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8424.379964] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 8432.378945] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8432.542838] Lustre: lustre-OST0001: new disk, initializing [ 8432.546861] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8432.557124] Lustre: 204699:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8432.642142] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8432.648513] Lustre: Skipped 2 previous similar messages [ 8437.764505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8437.780452] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8437.827321] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8438.611607] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 8448.447297] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8458.881271] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8461.002943] Lustre: DEBUG MARKER: ADD OST3 [ 8465.862184] loop: module loaded [ 8465.968885] loop0: detected capacity change from 0 to 204800000 [ 8466.188044] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8466.216206] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8466.235656] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8472.408100] loop0: detected capacity change from 0 to 204800000 [ 8473.020152] Lustre: DEBUG MARKER: STOP OST3 [ 8477.557508] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8477.767953] Lustre: lustre-OST0003: new disk, initializing [ 8477.773082] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8477.783050] Lustre: 206696:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8479.594096] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8479.734591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8479.747175] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8479.794700] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8481.638381] LustreError: 206897:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8481.670786] Lustre: Failing over lustre-OST0003 [ 8481.918438] Lustre: server umount lustre-OST0003 complete [ 8484.273856] LustreError: 203877:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.203.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8484.361762] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 8484.845737] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8484.856988] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8484.875313] Lustre: Skipped 1 previous similar message [ 8493.188777] Lustre: DEBUG MARKER: START OST3 again [ 8500.356227] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8500.676625] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8500.682613] Lustre: Skipped 1 previous similar message [ 8500.699364] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8501.931735] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8502.105479] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8502.108159] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8508.123960] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 8529.231146] Lustre: Failing over lustre-OST0003 [ 8529.349410] Lustre: server umount lustre-OST0003 complete [ 8531.447725] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8531.461389] LustreError: 203877:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8531.470028] Lustre: Skipped 1 previous similar message [ 8531.489545] LustreError: 203877:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 8551.909938] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8551.920379] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8551.928889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8551.928898] Lustre: Skipped 2 previous similar messages [ 8551.967421] Lustre: Skipped 3 previous similar messages [ 8553.871032] Lustre: server umount lustre-MDT0000 complete [ 8557.348072] LustreError: 202374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787588117 with bad export cookie 5890976850335023768 [ 8557.349042] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8557.360990] LustreError: 202374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8562.146868] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8562.162659] Lustre: Skipped 4 previous similar messages [ 8563.763598] Lustre: server umount lustre-MDT0001 complete [ 8567.942557] Lustre: server umount lustre-OST0000 complete [ 8571.696224] Lustre: server umount lustre-OST0001 complete [ 8578.566571] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 8586.899387] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 8625.004459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8625.217813] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8625.246876] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8625.323887] Lustre: lustre-MDT0000: new disk, initializing [ 8625.399184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8630.347806] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 8642.945070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8643.125792] Lustre: 212161:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8643.191216] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8643.196930] Lustre: Skipped 1 previous similar message [ 8643.356425] Lustre: lustre-MDT0001: new disk, initializing [ 8643.520587] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8643.531505] Lustre: Skipped 1 previous similar message [ 8643.575789] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8643.593190] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8648.160370] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 8657.139995] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8664.402877] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8672.222921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8672.705391] Lustre: 214135:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8674.528595] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8674.750201] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8679.809080] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 8689.108876] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8694.238964] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8694.566342] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8699.355163] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8699.629359] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8705.508523] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8705.510177] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8705.518356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8705.521749] LustreError: Skipped 1 previous similar message [ 8705.539964] Lustre: Skipped 3 previous similar messages [ 8707.820784] Lustre: server umount lustre-OST0000 complete [ 8715.232963] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8717.342656] Lustre: server umount lustre-MDT0000 complete [ 8720.552670] LustreError: 212968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787588280 with bad export cookie 5890976850335026064 [ 8720.553452] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8720.563820] LustreError: 212968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8720.908956] Lustre: server umount lustre-MDT0001 complete [ 8724.188191] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 12:18:02 (1787588282) [ 8738.541448] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 8747.745659] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 8782.236789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8782.573223] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8782.619510] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8782.625041] Lustre: Skipped 1 previous similar message [ 8782.715715] Lustre: lustre-MDT0000: new disk, initializing [ 8782.719250] Lustre: Skipped 1 previous similar message [ 8782.851407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8782.865024] Lustre: Skipped 1 previous similar message [ 8787.731248] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 8799.532180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8799.638836] Lustre: 218741:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8799.956781] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8804.511397] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 8814.452778] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8821.047313] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8827.534042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8827.817475] Lustre: 220713:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8829.520852] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8834.465869] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 8843.564603] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8849.052974] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8849.248967] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8854.848614] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8855.106147] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8860.135340] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8860.137090] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8860.138037] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8860.138042] Lustre: Skipped 3 previous similar messages [ 8860.154061] Lustre: Skipped 3 previous similar messages [ 8871.391904] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8871.637701] Lustre: server umount lustre-OST0000 complete [ 8882.061996] Lustre: server umount lustre-MDT0000 complete [ 8885.728848] LustreError: 220088:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8885.758862] LustreError: 220088:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 8886.396977] LustreError: 219536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787588446 with bad export cookie 5890976850335026946 [ 8886.400191] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8886.419813] LustreError: 219536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8886.729953] Lustre: server umount lustre-MDT0001 complete [ 8896.282052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8902.382789] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 8911.362077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8911.779680] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8911.786652] Lustre: Skipped 5 previous similar messages [ 8916.320502] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 8924.365616] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8931.049258] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8938.325694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8945.077824] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 8953.088543] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8962.538744] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8962.542333] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8962.547275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8962.547283] Lustre: Skipped 8 previous similar messages [ 8962.557594] Lustre: Skipped 3 previous similar messages [ 8962.565647] LustreError: Skipped 2 previous similar messages [ 8964.610522] Lustre: server umount lustre-OST0000 complete [ 8973.951500] Lustre: server umount lustre-MDT0000 complete [ 8977.419621] LustreError: 222913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787588537 with bad export cookie 5890976850335027828 [ 8977.423539] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8977.433032] LustreError: 222913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8977.800342] Lustre: server umount lustre-MDT0001 complete [ 9092.081621] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 9095.222268] Key type lgssc unregistered [ 9095.624071] LNet: 226600:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9095.640238] LNetError: 226600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9095.671893] LNet: Removed LNI 192.168.203.104@tcp [ 9096.592151] Key type .llcrypt unregistered [ 9096.598599] Key type ._llcrypt unregistered [ 9122.387946] Key type ._llcrypt registered [ 9122.395893] Key type .llcrypt registered [ 9122.475711] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 9138.859431] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 9140.193949] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9140.215514] alg: No test for adler32 (adler32-zlib) [ 9141.378474] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 9141.665769] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 9143.359160] Key type lgssc registered [ 9144.413739] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9178.562966] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9178.583954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9179.825908] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9179.881273] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9179.974500] Lustre: lustre-MDT0000: new disk, initializing [ 9180.075322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9180.101143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9184.042311] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 9193.914199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9194.001398] Lustre: 229954:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9194.057479] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9194.060776] Lustre: Skipped 1 previous similar message [ 9194.132252] Lustre: lustre-MDT0001: new disk, initializing [ 9194.223028] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9194.265114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9194.273581] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9198.319597] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 9209.103708] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9215.344738] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9223.087842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9223.475586] Lustre: lustre-OST0000: new disk, initializing [ 9223.480350] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9223.487862] Lustre: 231925:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9223.624452] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9230.400574] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 9231.982697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9232.006329] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9232.116698] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9241.673709] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9247.194444] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9247.378229] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9251.819778] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9252.025407] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9257.958347] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9257.964449] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9257.978660] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9257.980738] Lustre: Skipped 1 previous similar message [ 9263.078115] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9263.084323] Lustre: Skipped 2 previous similar messages [ 9268.192574] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9268.198929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9268.209853] Lustre: Skipped 1 previous similar message [ 9268.352551] Lustre: server umount lustre-OST0000 complete [ 9273.312332] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9273.325179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9273.342207] Lustre: Skipped 1 previous similar message [ 9278.434400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9278.441571] Lustre: Skipped 1 previous similar message [ 9278.643572] Lustre: server umount lustre-MDT0000 complete [ 9282.514273] LustreError: 229945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787588842 with bad export cookie 11126927122480684577 [ 9282.523903] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9282.528701] LustreError: 229945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9282.948614] Lustre: server umount lustre-MDT0001 complete [ 9303.060496] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 9313.626033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9313.960625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9318.606954] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 9319.398825] LustreError: 234861:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9319.421875] LustreError: 234861:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9323.492294] LustreError: 234862:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9328.462469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9328.629125] LustreError: 234861:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9328.850927] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9334.350563] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 9338.410859] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9346.176030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9346.706252] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9354.060650] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 9363.157157] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9363.344262] Lustre: lustre-OST0001: new disk, initializing [ 9363.348622] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9363.354506] Lustre: 237170:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9363.444540] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9368.664146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9368.678415] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9368.847493] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9370.527825] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 9381.048825] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9389.093406] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9403.205487] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 12:29:21 (1787588961) [ 9405.925217] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9405.930069] LustreError: Skipped 1 previous similar message [ 9405.936903] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9405.951755] Lustre: Skipped 1 previous similar message [ 9405.957373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9410.017195] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9410.041077] Lustre: Skipped 1 previous similar message [ 9411.840204] Lustre: server umount lustre-MDT0000 complete [ 9415.154677] LustreError: 234856:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9415.189215] LustreError: 234856:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9415.932597] LustreError: 234841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787588975 with bad export cookie 11126927122480685459 [ 9415.940853] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9415.947521] LustreError: 234841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9416.423488] Lustre: server umount lustre-MDT0001 complete [ 9430.730456] Lustre: server umount lustre-OST0000 complete [ 9444.595286] Lustre: server umount lustre-OST0001 complete [ 9450.241985] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 9459.330125] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 9492.506891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9492.686433] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9492.713341] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9492.771166] Lustre: lustre-MDT0000: new disk, initializing [ 9492.820452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9492.832940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9496.764879] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 9507.011971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9507.150579] Lustre: 241114:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9507.203372] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9507.207749] Lustre: Skipped 1 previous similar message [ 9507.294040] Lustre: lustre-MDT0001: new disk, initializing [ 9507.456655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9507.469630] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9511.647790] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 9519.769883] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9525.103877] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9532.480117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9532.793859] Lustre: lustre-OST0000: new disk, initializing [ 9532.798177] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9532.813987] Lustre: 243082:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9534.486614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9534.493577] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9534.614508] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9539.700055] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 9549.615669] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9554.512901] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9554.755848] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9559.357285] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9559.577230] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9565.159688] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9565.171781] Lustre: Skipped 2 previous similar messages [ 9565.177098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9565.184631] Lustre: Skipped 5 previous similar messages [ 9567.512577] Lustre: server umount lustre-OST0000 complete [ 9573.856262] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9573.869633] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9577.431450] Lustre: server umount lustre-MDT0000 complete [ 9580.517728] LustreError: 241127:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9580.545938] LustreError: 241127:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9580.728019] LustreError: 241108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787589140 with bad export cookie 11126927122480687594 [ 9580.733154] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9580.744068] LustreError: 241108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9581.046481] Lustre: server umount lustre-MDT0001 complete [ 9589.470792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9589.855974] LustreError: 245301:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9589.949950] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9589.955483] Lustre: Skipped 2 previous similar messages [ 9594.187652] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 9602.580252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9607.759175] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 9615.012474] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9621.525574] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9628.803212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9635.268611] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 9642.420656] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9787.872714] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9787.884442] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9787.902064] Lustre: Skipped 1 previous similar message [ 9787.908500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9787.920659] Lustre: Skipped 3 previous similar messages [ 9793.728794] Lustre: server umount lustre-OST0000 complete [ 9797.602247] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9797.608947] LustreError: Skipped 1 previous similar message [ 9803.586177] Lustre: server umount lustre-MDT0000 complete [ 9806.305074] LustreError: 246032:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9806.322239] LustreError: 246032:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9806.942983] LustreError: 245281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787589366 with bad export cookie 11126927122480688476 [ 9806.947944] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9806.956196] LustreError: 245281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9807.576368] Lustre: server umount lustre-MDT0001 complete [ 9911.942412] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 9914.938627] Key type lgssc unregistered [ 9915.221087] LNet: 248913:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9915.231941] LNetError: 248913:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9916.261950] LNet: Removed LNI 192.168.203.104@tcp [ 9917.437157] Key type .llcrypt unregistered [ 9917.443114] Key type ._llcrypt unregistered [ 9947.483301] Key type ._llcrypt registered [ 9947.485617] Key type .llcrypt registered [ 9947.604136] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 9964.324650] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 9965.163549] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9965.220121] alg: No test for adler32 (adler32-zlib) [ 9966.274712] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 9966.510435] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 9968.240262] Key type lgssc registered [ 9969.381757] Lustre: Echo OBD driver; http://www.lustre.org/ [10013.585873] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [10022.219565] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10022.248198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10023.459968] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10023.493792] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10023.573327] Lustre: lustre-MDT0000: new disk, initializing [10023.645360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10023.676052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10027.482363] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10036.577929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10036.692990] Lustre: 252987:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10036.789242] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10036.805882] Lustre: Skipped 1 previous similar message [10036.918547] Lustre: lustre-MDT0001: new disk, initializing [10037.029470] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10037.091725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10037.101984] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10041.333207] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10045.719629] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10051.877215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10052.117181] Lustre: lustre-OST0000: new disk, initializing [10052.123427] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10052.130112] Lustre: 254616:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10052.221541] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10054.462871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10054.472599] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10054.622592] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10057.949587] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10068.543241] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10068.792268] Lustre: lustre-OST0001: new disk, initializing [10068.795927] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10068.802679] Lustre: 255490:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10068.947256] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10074.988428] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10076.774841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10076.785990] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10076.861292] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10086.761820] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10094.270448] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10107.874384] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10107.882056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10107.894988] Lustre: Skipped 2 previous similar messages [10107.905764] Lustre: Skipped 3 previous similar messages [10110.778570] Lustre: server umount lustre-MDT0000 complete [10114.293608] LustreError: 252979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787589674 with bad export cookie 12263844434450679713 [10114.301789] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10114.317175] LustreError: 252979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10114.702156] Lustre: server umount lustre-MDT0001 complete [10129.057735] Lustre: server umount lustre-OST0000 complete [10143.987843] Lustre: server umount lustre-OST0001 complete [10160.028890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10168.537569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10175.418382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10183.349837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10201.651906] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [10212.725738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10212.740149] Lustre: lustre-MDT0000: reset Object Index mappings [10213.047381] LustreError: 260501:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10213.065990] LustreError: 260501:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10213.119106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10218.098534] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10218.469373] LustreError: 260502:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10222.562238] LustreError: 260501:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10227.165140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10227.191193] Lustre: lustre-MDT0001: reset Object Index mappings [10227.576891] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10231.978356] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10235.445776] Lustre: 261644:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10242.478548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10248.803868] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10253.097802] LustreError: 262001:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10257.192393] LustreError: 261999:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10257.217151] LustreError: 261999:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10257.219942] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10257.357972] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10257.592345] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10257.603894] Lustre: Skipped 1 previous similar message [10263.042479] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10263.511447] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10270.677516] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10274.531825] Lustre: 263514:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10281.331611] Lustre: server umount lustre-MDT0000 complete [10283.492492] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10283.494366] LustreError: 260497:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10283.525419] Lustre: Skipped 3 previous similar messages [10283.571765] LustreError: 260497:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10285.157904] LustreError: 261246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787589845 with bad export cookie 12263844434450686503 [10285.161472] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10285.172361] LustreError: 261246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10285.543464] Lustre: server umount lustre-MDT0001 complete [10300.148527] Lustre: server umount lustre-OST0000 complete [10304.805723] Lustre: 250494:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787589848/real 1787589848] req@ffff9913bdd2a680 x1874423474938112/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787589864 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10304.842255] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10304.859812] Lustre: Skipped 1 previous similar message [10305.050521] Lustre: server umount lustre-OST0001 complete [10324.150895] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 12:44:42 (1787589882) [10339.388626] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [10350.359687] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [10386.429634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10386.711126] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10386.765090] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10386.869969] Lustre: lustre-MDT0000: new disk, initializing [10386.931570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10386.946039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10390.872966] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10401.849078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10401.996712] Lustre: 267768:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10402.093097] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10402.097117] Lustre: Skipped 1 previous similar message [10402.200985] Lustre: lustre-MDT0001: new disk, initializing [10402.339051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10402.351248] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10406.319824] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10415.917458] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10421.239529] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10428.094858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10428.415494] Lustre: lustre-OST0000: new disk, initializing [10428.418835] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10428.424598] Lustre: 269737:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10430.109896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10430.116431] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10430.276867] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10434.596388] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10443.568296] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10448.737591] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10449.063461] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10454.051325] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10454.258717] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10456.038787] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10456.041955] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10456.046750] LustreError: Skipped 1 previous similar message [10456.047403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10456.060894] Lustre: Skipped 1 previous similar message [10456.090104] Lustre: Skipped 1 previous similar message [10461.167081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10461.177373] Lustre: Skipped 1 previous similar message [10466.287185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10466.296389] Lustre: Skipped 1 previous similar message [10470.367158] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10470.637181] Lustre: server umount lustre-OST0000 complete [10476.513945] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10476.517030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10476.540183] Lustre: Skipped 1 previous similar message [10476.560483] Lustre: Skipped 1 previous similar message [10486.763156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10486.770526] Lustre: Skipped 3 previous similar messages [10489.311118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10489.496788] Lustre: server umount lustre-MDT0000 complete [10491.871971] LustreError: 268646:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10491.901500] LustreError: 268646:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10492.593851] LustreError: 267760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787590052 with bad export cookie 12263844434450688596 [10492.596531] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10492.606791] LustreError: 267760:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10492.920981] Lustre: server umount lustre-MDT0001 complete [10510.778911] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [10520.963910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10521.549113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10521.563444] Lustre: Skipped 2 previous similar messages [10525.694867] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10526.694519] LustreError: 272674:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10526.709457] LustreError: 272674:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10534.671888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10539.549934] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10542.259917] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10547.883927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10554.106708] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10561.726495] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10561.969407] Lustre: lustre-OST0001: new disk, initializing [10561.972926] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10561.994799] Lustre: 274982:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10563.093389] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10563.105107] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10563.199890] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10568.136566] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10577.595262] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10581.527805] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10587.035684] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10587.066224] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10596.840698] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10596.852459] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10596.877910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10600.650435] Lustre: server umount lustre-MDT0000 complete [10602.466972] LustreError: 273719:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10602.511434] LustreError: 273719:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10604.428874] LustreError: 274981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787590164 with bad export cookie 12263844434450689478 [10604.441604] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10604.451198] LustreError: 274981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10604.798101] Lustre: server umount lustre-MDT0001 complete [10618.445835] Lustre: server umount lustre-OST0000 complete [10631.594404] Lustre: server umount lustre-OST0001 complete [10649.895487] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [10661.776032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10662.379973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10662.397910] Lustre: Skipped 3 previous similar messages [10668.495946] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10680.065926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10685.605937] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10688.532886] Lustre: 279182:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10695.150543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10702.408219] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10707.900891] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10711.790907] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10717.178148] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10719.103912] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10727.219220] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10731.608969] Lustre: 281060:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10737.081302] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10737.176799] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10746.648378] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 12:51:44 (1787590304) [10762.216829] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10762.230482] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10762.245806] Lustre: Skipped 3 previous similar messages [10762.257739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10762.266179] Lustre: Skipped 4 previous similar messages [10764.387088] Lustre: server umount lustre-MDT0000 complete [10768.285542] LustreError: 278023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787590328 with bad export cookie 12263844434450696016 [10768.293932] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10768.299603] LustreError: 278023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10768.354478] LustreError: 278038:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10768.385862] LustreError: 278038:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [10775.000483] Lustre: server umount lustre-MDT0001 complete [10785.504687] Lustre: server umount lustre-OST0000 complete [10790.417590] Lustre: server umount lustre-OST0001 complete [10798.110297] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [10810.297956] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [10866.564667] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [10877.542243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10877.777555] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10877.796085] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10877.862970] Lustre: lustre-MDT0000: new disk, initializing [10877.953670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10882.949939] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10894.275414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10894.428589] Lustre: 285300:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10894.724282] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10899.939890] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10915.716713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10915.917709] Lustre: 286937:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10918.043262] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10923.001612] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10935.084985] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10935.331982] Lustre: 287808:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10935.475632] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10935.480804] Lustre: Skipped 6 previous similar messages [10937.339413] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10937.348647] Lustre: Skipped 1 previous similar message [10937.442729] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10942.369232] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [10952.799412] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10956.926840] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10956.929747] Lustre: Skipped 1 previous similar message [10974.660816] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 12:55:32 (1787590532) [10978.287336] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10978.302215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10978.315430] Lustre: Skipped 8 previous similar messages [10978.334996] Lustre: Skipped 11 previous similar messages [10979.637689] Lustre: server umount lustre-MDT0000 complete [10983.162167] LustreError: 286098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787590543 with bad export cookie 12263844434450703660 [10983.169643] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10983.172289] LustreError: 286098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10983.682871] Lustre: server umount lustre-MDT0001 complete [10990.101603] Lustre: server umount lustre-OST0000 complete [10994.725742] Lustre: server umount lustre-OST0001 complete [11004.395887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11010.073768] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [11020.295307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11025.959628] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [11034.910520] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11041.547155] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11049.602515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11057.413354] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [11065.575220] LustreError: 291881:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11065.613315] LustreError: 291881:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [11066.155767] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11097.986241] loop1: detected capacity change from 0 to 204800000 [11098.338105] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11098.377146] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11098.415020] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11104.707834] loop1: detected capacity change from 0 to 204800000 [11111.521366] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11111.666116] LustreError: 290038:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs24410: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11111.701231] LustreError: 290038:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs24410 [11111.707905] LustreError: 290038:0:(mgs_handler.c:517:mgs_target_reg()) fs24410-MDT0001: Error checking target index: rc = -13 [11111.725575] LustreError: fs24410-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11111.738344] LustreError: 293793:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [11111.753453] LustreError: 293793:0:(tgt_mount.c:2132:server_put_super()) no obd fs24410-MDT0001 [11111.760297] LustreError: 293793:0:(tgt_mount.c:116:server_deregister_mount()) fs24410-MDT0001 not registered [11111.796921] Lustre: server umount fs24410-MDT0001 complete [11111.799167] LustreError: 293793:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11121.187385] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11121.354734] Lustre: Setting parameter fs24410-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs24410-MDT0001 [11121.400552] Lustre: srv-fs24410-MDT0001: No data found on store. Initialize space. [11121.410210] Lustre: Skipped 4 previous similar messages [11121.534686] Lustre: fs24410-MDT0001: new disk, initializing [11121.542376] Lustre: Skipped 3 previous similar messages [11121.731759] Lustre: srv-fs24410-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11121.741141] Lustre: 294259:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs24410-MDT0001-mdtlov: can't create new llog: rc = -115 [11121.750508] LustreError: 294259:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs24410-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11126.402275] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [11133.318351] Lustre: server umount fs24410-MDT0001 complete [11144.691377] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11148.134371] loop1: detected capacity change from 0 to 204800000 [11149.824827] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11155.807384] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11156.288554] Lustre: 295826:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs24410-MDT0001-mdtlov: can't create new llog: rc = -115 [11156.307805] LustreError: 295826:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs24410-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11161.571949] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [11166.963709] Lustre: server umount fs24410-MDT0001 complete [11179.493232] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11179.504671] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11179.522459] Lustre: Skipped 2 previous similar messages [11179.529384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11184.051699] Lustre: server umount lustre-MDT0000 complete [11193.878136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11194.119881] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11199.609777] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [11199.987694] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11209.378761] loop1: detected capacity change from 0 to 204800000 [11215.855542] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11216.408584] Lustre: 298273:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs24410-MDT0001-mdtlov: can't create new llog: rc = -115 [11216.419388] LustreError: 298273:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs24410-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11221.889659] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [11228.512481] Lustre: server umount fs24410-MDT0001 complete [11255.267469] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11259.497354] Lustre: server umount lustre-OST0000 complete [11269.831616] Lustre: server umount lustre-MDT0000 complete [11273.809320] LustreError: 290792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787590833 with bad export cookie 12263844434450706341 [11273.811938] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11273.818406] LustreError: 290792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11274.241913] Lustre: server umount lustre-MDT0001 complete [11286.154035] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [11289.699329] Key type lgssc unregistered [11290.023954] LNet: 300621:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11290.041932] LNetError: 300621:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11290.081192] LNet: Removed LNI 192.168.203.104@tcp [11291.109191] Key type .llcrypt unregistered [11291.114410] Key type ._llcrypt unregistered [11313.310659] Key type ._llcrypt registered [11313.318621] Key type .llcrypt registered [11313.532519] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [11315.239813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11315.349180] alg: No test for adler32 (adler32-zlib) [11316.627330] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [11317.033671] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [11318.776402] Key type lgssc registered [11320.207139] Lustre: Echo OBD driver; http://www.lustre.org/ [11328.826847] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11328.873096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11330.336835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11335.635034] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [11340.777556] LustreError: 301904:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11340.802827] LustreError: 301904:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11344.208694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11344.725257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11349.713619] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [11357.771537] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11363.883770] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11371.441260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11371.954761] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11376.049496] LustreError: 303734:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11378.674072] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [11381.254244] LustreError: 303734:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11386.350399] LustreError: 303736:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11386.745973] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11394.552631] LustreError: 303736:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11394.580491] LustreError: 303736:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11404.788716] LustreError: 303736:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11404.830977] LustreError: 303736:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11407.665271] loop1: detected capacity change from 0 to 204800000 [11407.973587] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11407.993646] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11408.025278] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11415.069049] loop1: detected capacity change from 0 to 204800000 [11422.204483] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11422.431921] Lustre: fs5337-OST0000: new disk, initializing [11422.440153] Lustre: srv-fs5337-OST0000: No data found on store. Initialize space. [11422.447538] Lustre: 305641:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11422.568674] Lustre: fs5337-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11425.265372] LustreError: 304368:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11425.300447] LustreError: 304368:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [11429.368393] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [11437.988618] Lustre: server umount fs5337-OST0000 complete [11451.209758] loop1: detected capacity change from 0 to 204800000 [11455.993488] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11457.572468] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11457.976671] Lustre: fs5337-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11461.115374] LustreError: 303735:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11461.145672] LustreError: 303735:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 20 previous similar messages [11464.536366] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [11471.339196] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11475.461113] loop2: detected capacity change from 0 to 204800000 [11475.741615] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11475.776313] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11475.793297] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11482.053409] loop2: detected capacity change from 0 to 204800000 [11488.022287] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11488.102774] LustreError: fs5337-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11488.114888] LustreError: fs5337-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11488.131464] LustreError: 308682:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [11488.157445] LustreError: 308682:0:(tgt_mount.c:2132:server_put_super()) no obd fs5337-OST0001 [11488.167187] LustreError: 308682:0:(tgt_mount.c:116:server_deregister_mount()) fs5337-OST0001 not registered [11488.213436] Lustre: server umount fs5337-OST0001 complete [11488.217747] LustreError: 308682:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11498.910995] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11499.108706] Lustre: fs5337-OST0001: new disk, initializing [11499.112400] Lustre: srv-fs5337-OST0001: No data found on store. Initialize space. [11499.118922] Lustre: 309232:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11499.181191] Lustre: fs5337-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11505.649581] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [11514.655239] Lustre: server umount fs5337-OST0000 complete [11526.113971] Lustre: server umount fs5337-OST0001 complete [11527.666476] LustreError: 303734:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11527.675792] LustreError: 303734:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [11541.017258] loop1: detected capacity change from 0 to 204800000 [11547.048752] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11547.365957] Lustre: fs5337-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11554.894534] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [11563.800637] loop2: detected capacity change from 0 to 204800000 [11571.322391] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11580.096960] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [11591.500612] Lustre: server umount fs5337-OST0000 complete [11602.444868] Lustre: server umount fs5337-OST0001 complete [11609.584214] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11619.811808] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11619.840725] Lustre: Skipped 1 previous similar message [11619.850744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11624.922587] Lustre: server umount lustre-OST0000 complete [11630.048592] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11630.065395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11630.065868] Lustre: Skipped 1 previous similar message [11630.081365] Lustre: Skipped 2 previous similar messages [11635.169368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11635.178574] Lustre: Skipped 1 previous similar message [11639.267975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11639.280476] Lustre: Skipped 1 previous similar message [11644.386412] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11644.393183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11644.405588] Lustre: Skipped 1 previous similar message [11644.535341] Lustre: server umount lustre-MDT0000 complete [11649.369213] LustreError: 301885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787591209 with bad export cookie 13327991568855143760 [11649.389555] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11649.397370] LustreError: 301885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11649.985316] Lustre: server umount lustre-MDT0001 complete [11663.275993] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [11666.581976] Key type lgssc unregistered [11666.954748] LNet: 315161:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11666.965823] LNetError: 315161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11666.996378] LNet: Removed LNI 192.168.203.104@tcp [11668.065177] Key type .llcrypt unregistered [11668.067049] Key type ._llcrypt unregistered [11708.370829] Key type ._llcrypt registered [11708.374835] Key type .llcrypt registered [11708.520203] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [11726.874605] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [11727.952129] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11728.021141] alg: No test for adler32 (adler32-zlib) [11729.331424] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [11729.701568] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [11731.463186] Key type lgssc registered [11732.716857] Lustre: Echo OBD driver; http://www.lustre.org/ [11774.645863] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11774.695794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11776.128610] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11776.187387] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11776.291811] Lustre: lustre-MDT0000: new disk, initializing [11776.464849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11776.495239] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11781.598968] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [11794.313211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11794.396326] Lustre: 319023:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11794.440894] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11794.446617] Lustre: Skipped 1 previous similar message [11794.496876] Lustre: lustre-MDT0001: new disk, initializing [11794.622268] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11794.679450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11794.688936] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11801.301135] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [11814.067695] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11822.123478] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11831.101946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11831.452301] Lustre: lustre-OST0000: new disk, initializing [11831.459588] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11831.468908] Lustre: 320996:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11831.605177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11838.511309] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [11839.037959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11839.050775] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11839.207784] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11848.680475] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11850.916845] Lustre: Mounted lustre-client - version 2.17.57_80_g35a8f2d [11852.178832] Lustre: Unmounted lustre-client [11861.476846] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11861.487707] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11861.495071] Lustre: Skipped 1 previous similar message [11866.593991] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11866.602325] Lustre: Skipped 1 previous similar message [11867.596252] Lustre: server umount lustre-OST0000 complete [11871.713900] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11871.718100] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11871.726468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11871.742715] Lustre: Skipped 2 previous similar messages [11876.833296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11876.842019] Lustre: Skipped 2 previous similar messages [11877.831589] Lustre: server umount lustre-MDT0000 complete [11882.473762] LustreError: 319016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787591442 with bad export cookie 8971117223525236839 [11882.478127] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11882.489193] LustreError: 319016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11882.717754] Lustre: server umount lustre-MDT0001 complete [11897.781663] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [11901.260714] Key type lgssc unregistered [11911.841669] LNet: 323159:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11911.856958] LNetError: 323159:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11982.511844] LNet: Removed LNI 192.168.203.104@tcp [11983.694430] Key type .llcrypt unregistered [11983.698382] Key type ._llcrypt unregistered [12005.111709] Key type ._llcrypt registered [12005.114126] Key type .llcrypt registered [12005.217326] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [12021.810598] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [12023.158382] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12023.206370] alg: No test for adler32 (adler32-zlib) [12024.195387] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [12024.432741] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [12026.081708] Key type lgssc registered [12027.361871] Lustre: Echo OBD driver; http://www.lustre.org/ [12062.920835] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 13:13:40 (1787591620) [12070.817988] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12070.885410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12072.115722] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12072.140216] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12072.209754] Lustre: lustre-MDT0000: new disk, initializing [12072.306352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12072.316564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12077.818651] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [12089.786429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12090.022327] Lustre: 326384:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12090.200845] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12090.208272] Lustre: Skipped 1 previous similar message [12090.428156] Lustre: lustre-MDT0001: new disk, initializing [12090.626289] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12090.730556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12090.764927] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12096.832887] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [12109.654654] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12116.210535] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12124.807301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12125.204590] Lustre: lustre-OST0000: new disk, initializing [12125.208412] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12125.222878] Lustre: 328357:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12125.375711] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12130.918740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12130.938102] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12131.036287] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12132.647342] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [12142.380384] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12162.017771] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12162.022061] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12162.023212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12162.050113] Lustre: Skipped 1 previous similar message [12167.139414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12167.170433] Lustre: Skipped 2 previous similar messages [12167.911926] Lustre: server umount lustre-OST0000 complete [12177.376880] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12177.392560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12179.753859] Lustre: server umount lustre-MDT0000 complete [12183.411706] LustreError: 326378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787591743 with bad export cookie 10552823013951034265 [12183.419161] LustreError: 326378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12183.421272] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12183.795541] Lustre: server umount lustre-MDT0001 complete [12196.273191] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [12199.718396] Key type lgssc unregistered [12200.091231] LNet: 330274:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12200.103730] LNetError: 330274:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12200.126507] LNet: Removed LNI 192.168.203.104@tcp [12201.189356] Key type .llcrypt unregistered [12201.197435] Key type ._llcrypt unregistered [12223.083575] Key type ._llcrypt registered [12223.087644] Key type .llcrypt registered [12223.207421] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [12224.250420] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12224.320594] alg: No test for adler32 (adler32-zlib) [12225.389649] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [12225.687964] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [12227.376366] Key type lgssc registered [12228.647907] Lustre: Echo OBD driver; http://www.lustre.org/ [12236.881247] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12236.904430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12238.295572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12243.462563] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [12248.544775] LustreError: 331550:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12248.562653] LustreError: 331550:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12252.974446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12253.425199] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12257.786831] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [12266.014547] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12272.621161] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12280.846563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12281.228308] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12288.505938] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [12297.586631] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12307.387640] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12309.006933] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12314.082183] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12314.092754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12314.103689] Lustre: Skipped 2 previous similar messages [12314.128482] Lustre: Skipped 2 previous similar messages [12315.047541] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.4@tcp (stopping) [12319.202928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12324.322331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12324.325538] Lustre: Skipped 3 previous similar messages [12327.391595] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12327.794251] Lustre: server umount lustre-MDT0000 complete [12329.442870] LustreError: 331549:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12329.476720] LustreError: 331549:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12332.620463] LustreError: 331530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787591892 with bad export cookie 12748575108854736842 [12332.630120] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12332.643272] LustreError: 331530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12334.561284] LustreError: 331544:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12334.562974] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12334.586714] LustreError: 331544:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12334.612794] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12334.643709] Lustre: Skipped 1 previous similar message [12337.568684] LustreError: 331544:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12337.586346] LustreError: 331544:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12339.328233] Lustre: server umount lustre-MDT0001 complete [12344.883136] Lustre: server umount lustre-OST0000 complete [12357.726174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12363.691509] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [12367.050890] Lustre: 335732:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.104@tcp [12369.615440] Lustre: server umount MGS complete [12378.619074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12379.165157] LustreError: 336307:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12379.203152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12383.938146] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [12389.344271] LustreError: 336308:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12389.360880] LustreError: 336308:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12392.716739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12393.160495] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12398.086438] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [12407.601813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12408.119160] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12415.279140] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [12424.151452] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12425.891397] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12432.520641] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12434.196526] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12440.055821] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [12440.078109] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12440.119833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12440.128359] Lustre: Skipped 2 previous similar messages [12441.827368] Lustre: server umount lustre-OST0000 complete [12446.178977] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12446.203482] Lustre: Skipped 1 previous similar message [12452.311627] Lustre: server umount lustre-MDT0000 complete [12456.420498] LustreError: 336307:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12456.445565] LustreError: 336307:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12456.689075] LustreError: 336289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787592016 with bad export cookie 12748575108854737682 [12456.697710] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12456.707681] LustreError: 336289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12457.296077] Lustre: server umount lustre-MDT0001 complete [12469.050556] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [12471.647130] Key type lgssc unregistered [12471.910493] LNet: 339745:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12471.917777] LNetError: 339745:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12471.936099] LNet: Removed LNI 192.168.203.104@tcp [12472.748877] Key type .llcrypt unregistered [12472.755471] Key type ._llcrypt unregistered [12491.596740] Key type ._llcrypt registered [12491.599196] Key type .llcrypt registered [12491.730543] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [12507.911294] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [12509.520059] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12509.547044] alg: No test for adler32 (adler32-zlib) [12510.652828] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [12510.858174] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [12512.543169] Key type lgssc registered [12513.937649] Lustre: Echo OBD driver; http://www.lustre.org/ [12554.359818] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12554.417629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12555.805616] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12555.831260] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12555.923832] Lustre: lustre-MDT0000: new disk, initializing [12556.004352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12556.025450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12561.094588] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [12572.914165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12573.102873] Lustre: 342906:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12573.214292] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12573.218406] Lustre: Skipped 1 previous similar message [12573.313681] Lustre: lustre-MDT0001: new disk, initializing [12573.427983] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12573.501134] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12573.522061] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12578.912436] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [12589.159558] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12594.928409] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12601.741310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12601.978206] Lustre: lustre-OST0000: new disk, initializing [12601.981579] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12601.985686] Lustre: 344878:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12602.071804] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12608.763794] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [12612.171600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12612.182750] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12612.360353] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12618.479577] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12623.218263] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12623.438420] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12628.059235] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12628.263961] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12633.056743] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12633.059688] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12633.070263] LustreError: Skipped 1 previous similar message [12633.090492] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12633.113294] Lustre: Skipped 1 previous similar message [12636.458724] Lustre: server umount lustre-OST0000 complete [12640.238612] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12640.244103] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12640.259872] Lustre: Skipped 1 previous similar message [12640.266505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12643.296905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12643.300117] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12646.677505] Lustre: server umount lustre-MDT0000 complete [12650.882701] LustreError: 342898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787592210 with bad export cookie 8622041251569853572 [12650.899114] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12650.907343] LustreError: 342898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12651.326831] Lustre: server umount lustre-MDT0001 complete [12659.844165] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 13:23:37 (1787592217) [12675.060678] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [12685.049445] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [12717.901156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12718.225754] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12718.274448] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12718.402739] Lustre: lustre-MDT0000: new disk, initializing [12718.524189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12718.566896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12723.662574] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [12736.673631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12736.828336] Lustre: 349638:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12736.944103] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12736.953511] Lustre: Skipped 1 previous similar message [12737.094342] Lustre: lustre-MDT0001: new disk, initializing [12737.210160] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12737.239448] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12737.250366] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12742.039299] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [12751.170898] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12755.936771] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12762.368494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12762.578598] Lustre: lustre-OST0000: new disk, initializing [12762.582476] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12762.589563] Lustre: 351607:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12762.667646] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12768.552292] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [12771.409596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12771.425799] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12771.538928] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12777.107316] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12807.651511] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12807.655392] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12807.673639] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12807.691834] Lustre: Skipped 2 previous similar messages [12811.717893] Lustre: server umount lustre-OST0000 complete [12817.890541] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12817.892610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12817.909074] Lustre: Skipped 2 previous similar messages [12817.919853] Lustre: Skipped 1 previous similar message [12821.694620] Lustre: server umount lustre-MDT0000 complete [12823.008767] LustreError: 349647:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12823.021485] LustreError: 349647:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12825.448341] LustreError: 349632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787592385 with bad export cookie 8622041251569854454 [12825.454810] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12825.458799] LustreError: 349632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12825.829159] Lustre: server umount lustre-MDT0001 complete [12836.745267] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [12840.146121] Key type lgssc unregistered [12840.619691] LNet: 353764:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12840.627862] LNetError: 353764:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12840.686391] LNet: Removed LNI 192.168.203.104@tcp [12841.603552] Key type .llcrypt unregistered [12841.605452] Key type ._llcrypt unregistered [12940.405414] Key type ._llcrypt registered [12940.409551] Key type .llcrypt registered [12940.522908] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [12940.892410] Key type .llcrypt unregistered [12940.898570] Key type ._llcrypt unregistered [13014.308487] Key type ._llcrypt registered [13014.313967] Key type .llcrypt registered [13014.430752] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [13014.710857] Key type .llcrypt unregistered [13014.713900] Key type ._llcrypt unregistered [13042.903332] Key type ._llcrypt registered [13042.915459] Key type .llcrypt registered [13043.072259] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [13043.427270] Key type .llcrypt unregistered [13043.428992] Key type ._llcrypt unregistered [13076.874535] Key type ._llcrypt registered [13076.878093] Key type .llcrypt registered [13077.028461] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [13077.369543] Key type .llcrypt unregistered [13077.376235] Key type ._llcrypt unregistered [13153.192717] Key type ._llcrypt registered [13153.198167] Key type .llcrypt registered [13153.326570] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [13153.625543] Key type .llcrypt unregistered [13153.627656] Key type ._llcrypt unregistered [13184.315298] Key type ._llcrypt registered [13184.317972] Key type .llcrypt registered [13184.421943] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [13184.742679] Key type .llcrypt unregistered [13184.748371] Key type ._llcrypt unregistered [13211.722653] Key type ._llcrypt registered [13211.727397] Key type .llcrypt registered [13211.859259] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [13212.128637] Key type .llcrypt unregistered [13212.134803] Key type ._llcrypt unregistered [13238.984146] Key type ._llcrypt registered [13238.987398] Key type .llcrypt registered [13239.098882] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [13239.383339] Key type .llcrypt unregistered [13239.385977] Key type ._llcrypt unregistered [13268.494615] Key type ._llcrypt registered [13268.496759] Key type .llcrypt registered [13268.575612] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [13268.888588] Key type .llcrypt unregistered [13268.890596] Key type ._llcrypt unregistered [13298.549918] Key type ._llcrypt registered [13298.552296] Key type .llcrypt registered [13298.619341] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [13298.883630] Key type .llcrypt unregistered [13298.887803] Key type ._llcrypt unregistered [13365.548373] Key type ._llcrypt registered [13365.550328] Key type .llcrypt registered [13365.634325] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [13365.992646] Key type .llcrypt unregistered [13365.999747] Key type ._llcrypt unregistered [13397.847506] Key type ._llcrypt registered [13397.852571] Key type .llcrypt registered [13397.985972] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [13398.363440] Key type .llcrypt unregistered [13398.367094] Key type ._llcrypt unregistered [13430.874692] Key type ._llcrypt registered [13430.878466] Key type .llcrypt registered [13431.058046] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [13431.591468] Key type .llcrypt unregistered [13431.594990] Key type ._llcrypt unregistered [13490.639553] Key type ._llcrypt registered [13490.642463] Key type .llcrypt registered [13490.763773] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [13503.933501] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [13504.770070] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13504.817431] alg: No test for adler32 (adler32-zlib) [13505.873160] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [13506.083977] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [13507.720656] Key type lgssc registered [13508.811611] Lustre: Echo OBD driver; http://www.lustre.org/ [13538.350120] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13538.394859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13539.655285] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13539.702095] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13539.796859] Lustre: tst250fs-MDT0000: new disk, initializing [13539.873505] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13539.926479] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13544.475907] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [13554.272588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13554.396691] Lustre: 374054:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13554.474435] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13554.480173] Lustre: Skipped 1 previous similar message [13554.546577] Lustre: tst250fs-MDT0001: new disk, initializing [13554.689412] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13554.742571] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13554.757398] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13559.155357] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [13563.961834] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13569.894650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13570.193876] Lustre: tst250fs-OST0000: new disk, initializing [13570.200924] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13570.215633] Lustre: 375685:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13570.309048] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13576.261353] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13576.277579] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13576.414616] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13576.483663] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [13586.275137] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13586.509545] Lustre: tst250fs-OST0001: new disk, initializing [13586.513807] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13586.519446] Lustre: 376556:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13586.612517] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13592.391098] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [13594.680838] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13594.689248] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13594.783935] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13597.303695] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13599.564360] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [13599.572350] Lustre: Skipped 2 previous similar messages [13620.705701] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13620.717688] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13620.743454] Lustre: Skipped 1 previous similar message [13620.768329] Lustre: Skipped 3 previous similar messages [13623.215444] Lustre: server umount tst250fs-MDT0000 complete [13627.184319] LustreError: 374046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787593187 with bad export cookie 5039506427819320723 [13627.189932] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13627.193878] LustreError: 374046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13627.543197] Lustre: server umount tst250fs-MDT0001 complete [13642.283855] Lustre: server umount tst250fs-OST0000 complete [13656.480440] Lustre: server umount tst250fs-OST0001 complete [13673.965440] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [13684.622361] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [13716.570493] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 13:41:14 (1787593274) [13718.102347] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13733.810906] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [13743.457303] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [13781.085457] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13520 sec ========= 13:42:18 (1787593338) [13782.772825] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 13:42:20 (1787593340) === [13786.909821] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 13:42:24 (1787593344) === [13828.381921] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [13831.607116] Key type lgssc unregistered [13831.940737] LNet: 385690:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13831.949157] LNetError: 385690:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13831.963415] LNet: Removed LNI 192.168.203.104@tcp [13832.823336] Key type .llcrypt unregistered [13832.826958] Key type ._llcrypt unregistered