[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 1116420105 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002417] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004017] kvm-guest: setup PV IPIs [ 0.008492] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.011008] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.012026] pid_max: default: 32768 minimum: 301 [ 0.014139] LSM: Security Framework initializing [ 0.015058] Yama: becoming mindful. [ 0.016032] SELinux: Initializing. [ 0.017100] *** VALIDATE selinux *** [ 0.027423] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.031000] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.031216] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033132] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.035091] *** VALIDATE tmpfs *** [ 0.037228] *** VALIDATE proc *** [ 0.038350] *** VALIDATE cgroup *** [ 0.039019] *** VALIDATE cgroup2 *** [ 0.041231] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.043175] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.044018] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.045036] Spectre V2 : User space: Vulnerable [ 0.046018] Speculative Store Bypass: Vulnerable [ 0.050077] debug: unmapping init [mem 0xffffffff9de59000-0xffffffff9de60fff] [ 0.054480] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.056860] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.057036] ... version: 2 [ 0.058016] ... bit width: 48 [ 0.059027] ... generic registers: 4 [ 0.060032] ... value mask: 0000ffffffffffff [ 0.061038] ... max period: 00007fffffffffff [ 0.062035] ... fixed-purpose events: 3 [ 0.063029] ... event mask: 000000070000000f [ 0.065070] rcu: Hierarchical SRCU implementation. [ 0.071796] smp: Bringing up secondary CPUs ... [ 0.073803] x86: Booting SMP configuration: [ 0.074049] .... node #0, CPUs: #1 #2 #3 [ 0.088176] smp: Brought up 1 node, 4 CPUs [ 0.090021] smpboot: Max logical packages: 1 [ 0.091022] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.169130] node 0 deferred pages initialised in 76ms [ 0.179162] devtmpfs: initialized [ 0.181807] x86/mm: Memory block size: 128MB [ 0.218000] gcov: version magic: 0x41383552 [ 0.225439] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.229188] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.230370] pinctrl core: initialized pinctrl subsystem [ 0.232232] [ 0.232914] ************************************************************* [ 0.234021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.237073] ** ** [ 0.239012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.241029] ** ** [ 0.242000] ** This means that this kernel is built to expose internal ** [ 0.242013] ** IOMMU data structures, which may compromise security on ** [ 0.245014] ** your system. ** [ 0.248015] ** ** [ 0.250012] ** If you see this message and you are not debugging the ** [ 0.252015] ** kernel, report this immediately to your vendor! ** [ 0.254016] ** ** [ 0.256018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.258014] ************************************************************* [ 0.260052] NET: Registered protocol family 16 [ 0.262488] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.265064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.268079] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.273025] cpuidle: using governor menu [ 0.293431] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.297089] PCI: Using configuration type 1 for base access [ 0.301162] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.344110] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.348021] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.423109] cryptd: max_cpu_qlen set to 1000 [ 0.425297] ACPI: Added _OSI(Module Device) [ 0.426017] ACPI: Added _OSI(Processor Device) [ 0.428000] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.429020] ACPI: Added _OSI(Processor Aggregator Device) [ 0.442066] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.471756] ACPI: Interpreter enabled [ 0.472000] ACPI: PM: (supports S0 S3 S4 S5) [ 0.472000] ACPI: Using IOAPIC for interrupt routing [ 0.472000] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.472000] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.472000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.472000] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.472000] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.472000] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.472000] acpiphp: Slot [2] registered [ 0.472000] acpiphp: Slot [5] registered [ 0.472000] acpiphp: Slot [6] registered [ 0.472000] acpiphp: Slot [7] registered [ 0.472000] acpiphp: Slot [8] registered [ 0.472000] acpiphp: Slot [9] registered [ 0.472000] acpiphp: Slot [10] registered [ 0.472000] acpiphp: Slot [3] registered [ 0.472000] acpiphp: Slot [4] registered [ 0.472000] acpiphp: Slot [11] registered [ 0.472000] acpiphp: Slot [12] registered [ 0.472000] acpiphp: Slot [13] registered [ 0.472000] acpiphp: Slot [14] registered [ 0.472000] acpiphp: Slot [15] registered [ 0.472000] acpiphp: Slot [16] registered [ 0.472000] acpiphp: Slot [17] registered [ 0.472000] acpiphp: Slot [18] registered [ 0.472000] acpiphp: Slot [19] registered [ 0.472000] acpiphp: Slot [20] registered [ 0.472000] acpiphp: Slot [21] registered [ 0.472000] acpiphp: Slot [22] registered [ 0.472000] acpiphp: Slot [23] registered [ 0.472000] acpiphp: Slot [24] registered [ 0.472000] acpiphp: Slot [25] registered [ 0.472000] acpiphp: Slot [26] registered [ 0.472000] acpiphp: Slot [27] registered [ 0.472000] acpiphp: Slot [28] registered [ 0.472000] acpiphp: Slot [29] registered [ 0.472000] acpiphp: Slot [30] registered [ 0.472000] acpiphp: Slot [31] registered [ 0.472000] PCI host bridge to bus 0000:00 [ 0.472000] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.472000] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.472000] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.472000] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.472000] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.472000] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.472000] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.472000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.472000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.472000] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.472000] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.472000] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.472000] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.472000] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.472000] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.472000] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.472000] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.472000] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.472000] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.472000] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.472000] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.472000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.472000] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.472000] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.472000] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.472000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.472000] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.472000] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.472000] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.472000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.472000] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.472000] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.472000] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.472000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.476021] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.484010] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.556011] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.577314] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.594021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.611021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.637022] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.647181] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.654018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.662022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.683022] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.701182] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.703557] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.706504] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.710006] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.713265] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.719050] iommu: Default domain type: Passthrough [ 0.720596] SCSI subsystem initialized [ 0.721125] ACPI: bus type USB registered [ 0.722129] usbcore: registered new interface driver usbfs [ 0.723087] usbcore: registered new interface driver hub [ 0.724126] usbcore: registered new device driver usb [ 0.725193] pps_core: LinuxPPS API ver. 1 registered [ 0.726009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.727146] PTP clock support registered [ 0.728378] EDAC MC: Ver: 3.0.0 [ 0.729484] PCI: Using ACPI for IRQ routing [ 0.732150] NetLabel: Initializing [ 0.733011] NetLabel: domain hash size = 128 [ 0.735012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.738096] NetLabel: unlabeled traffic allowed by default [ 0.740449] vgaarb: loaded [ 0.742899] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.745016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.754192] clocksource: Switched to clocksource kvm-clock [ 0.893951] VFS: Disk quotas dquot_6.6.0 [ 0.897284] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.905711] *** VALIDATE ramfs *** [ 0.907107] *** VALIDATE hugetlbfs *** [ 0.908707] pnp: PnP ACPI init [ 0.911630] pnp: PnP ACPI: found 6 devices [ 0.939015] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.944155] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.946529] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.951771] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.954166] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.958607] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.965208] NET: Registered protocol family 2 [ 0.970172] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.980176] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.986872] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.998849] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.006517] TCP: Hash tables configured (established 65536 bind 65536) [ 1.013294] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.020623] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.027386] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.032224] NET: Registered protocol family 1 [ 1.035567] RPC: Registered named UNIX socket transport module. [ 1.037789] RPC: Registered udp transport module. [ 1.039504] RPC: Registered tcp transport module. [ 1.041362] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.043910] NET: Registered protocol family 44 [ 1.045721] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.047956] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.050075] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.052441] PCI: CLS 0 bytes, default 64 [ 1.054296] Unpacking initramfs... [ 2.996345] debug: unmapping init [mem 0xffff8add7cc54000-0xffff8add7ffbffff] [ 3.001124] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.003619] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.006962] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.716184] Initialise system trusted keyrings [ 3.717422] Key type blacklist registered [ 3.718751] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.726418] zbud: loaded [ 3.728760] *** VALIDATE nfs *** [ 3.729498] *** VALIDATE nfs4 *** [ 3.730566] pstore: using deflate compression [ 3.735736] Platform Keyring initialized [ 3.832595] NET: Registered protocol family 38 [ 3.833977] Key type asymmetric registered [ 3.835220] Asymmetric key parser 'x509' registered [ 3.836778] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.838940] io scheduler mq-deadline registered [ 3.840289] io scheduler kyber registered [ 3.841531] io scheduler bfq registered [ 3.843210] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.845476] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.847724] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.849772] ACPI: Power Button [PWRF] [ 3.855785] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.861792] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.877088] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.943036] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.962940] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.992748] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.022753] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.027754] Non-volatile memory driver v1.3 [ 4.029502] Linux agpgart interface v0.103 [ 4.061755] virtio_blk virtio1: [vda] 144832 512-byte logical blocks (74.2 MB/70.7 MiB) [ 4.066689] vda: detected capacity change from 0 to 74153984 [ 4.085461] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.092117] vdb: detected capacity change from 0 to 1073741824 [ 4.113741] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.116946] vdc: detected capacity change from 0 to 2621440000 [ 4.147746] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.152171] vdd: detected capacity change from 0 to 2621440000 [ 4.176604] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.181473] vde: detected capacity change from 0 to 4294967296 [ 4.212779] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.218443] vdf: detected capacity change from 0 to 4294967296 [ 4.236286] libphy: Fixed MDIO Bus: probed [ 4.246810] usbcore: registered new interface driver usbserial_generic [ 4.252801] usbserial: USB Serial support registered for generic [ 4.256790] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.264749] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.269906] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.273431] mousedev: PS/2 mouse device common for all mice [ 4.281473] rtc_cmos 00:05: RTC can wake from S4 [ 4.286086] rtc_cmos 00:05: registered as rtc0 [ 4.288495] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.291791] intel_pstate: CPU model not supported [ 4.292554] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.299102] hid: raw HID events driver (C) Jiri Kosina [ 4.304495] usbcore: registered new interface driver usbhid [ 4.307010] usbhid: USB HID core driver [ 4.308847] drop_monitor: Initializing network drop monitor service [ 4.312950] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.314420] Initializing XFRM netlink socket [ 4.314921] NET: Registered protocol family 10 [ 4.317377] Segment Routing with IPv6 [ 4.323518] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.325801] NET: Registered protocol family 17 [ 4.335506] mpls_gso: MPLS GSO support [ 4.341880] RAS: Correctable Errors collector initialized. [ 4.344522] AVX version of gcm_enc/dec engaged. [ 4.346647] AES CTR mode by8 optimization enabled [ 4.478841] sched_clock: Marking stable (4478657369, 0)->(7198867896, -2720210527) [ 4.483645] registered taskstats version 1 [ 4.487059] Loading compiled-in X.509 certificates [ 4.496448] zswap: loaded using pool lzo/zbud [ 4.527225] Key type big_key registered [ 4.543594] Key type encrypted registered [ 4.545733] ima: No TPM chip found, activating TPM-bypass! [ 4.548449] ima: Allocated hash algorithm: sha1 [ 4.550669] ima: No architecture policies found [ 4.552675] evm: Initialising EVM extended attributes: [ 4.555400] evm: security.selinux [ 4.556957] evm: security.ima [ 4.558609] evm: security.capability [ 4.560266] evm: HMAC attrs: 0x1 [ 4.563581] rtc_cmos 00:05: setting system clock to 2026-06-26 13:26:23 UTC (1782480383) [ 4.572282] debug: unmapping init [mem 0xffffffff9ee03000-0xffffffff9effffff] [ 4.575938] debug: unmapping init [mem 0xffffffff9db82000-0xffffffff9de58fff] [ 4.603449] Write protecting the kernel read-only data: 28672k [ 4.607819] debug: unmapping init [mem 0xffffffff9c203000-0xffffffff9c3fffff] [ 4.611713] debug: unmapping init [mem 0xffffffff9cb14000-0xffffffff9cbfffff] [ 4.681822] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.700553] systemd[1]: Detected virtualization kvm. [ 4.707143] systemd[1]: Detected architecture x86-64. [ 4.709874] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.844220] systemd[1]: No hostname configured. [ 4.846232] systemd[1]: Set hostname to . [ 4.850822] random: systemd: uninitialized urandom read (16 bytes read) [ 4.856494] systemd[1]: Initializing machine ID from random generator. [ 4.972340] random: ln: uninitialized urandom read (6 bytes read) [ 5.124252] random: systemd: uninitialized urandom read (16 bytes read) [ 5.128927] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 5.140156] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 5.159278] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.065750] device-mapper: uevent: version 1.0.3 [ 7.068445] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ 8.728764] virtio_net virtio0 ens2: renamed from eth0 Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK [ 9.188741] random: fast init done ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 11.044197] scsi host0: ata_piix [ 11.127866] scsi host1: ata_piix [ 11.132197] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 11.141547] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 16.132512] random: crng init done [ 16.136627] random: 7 urandom warning(s) missed due to ratelimiting [ 20.049899] dracut-initqueue[575]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 22.670800] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 26.833436] printk: systemd: 26 output lines suppressed due to ratelimiting [ 27.570566] SELinux: Disabled at runtime. [ 27.695035] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 27.709846] systemd[1]: Detected virtualization kvm. [ 27.715627] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 28.752980] hrtimer: interrupt took 6990919 ns [ 29.413843] systemd[1]: initrd-switch-root.service: Succeeded. [ 29.426970] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 29.464707] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 29.485572] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 29.498768] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 29.525280] systemd[1]: Starting Journal Service... Starting Journal Service... [ 29.575159] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on Process Core Dump Socket. Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. [ OK ] Stopped target Switch Root. Mounting POSIX Message Queue File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-getty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target rpc_pipefs.target. Mounting Kernel Debug File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Stopped target Initrd File Systems. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [ OK ] Reached target Slices. [ 30.443237] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 31.480593] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 32.955708] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 33.554124] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 34.458666] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 34.539134] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root support (10s / no limit) [ **] A start job is running for Configur…only root support (11s / no limit) [ *] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (12s / no limit) [ ***] A start job is running for Configur…only root support (12s / no limit) [ *** ] A start job is running for Configur…only root support (13s / no limit) [ *** ] A start job is running for Configur…only root support (14s / no limit) [*** ] A start job is running for Configur…only root support (14s / no limit) [** ] A start job is running for Configur…only root support (15s / no limit) [* ] A start job is running for Configur…only root support (15s / no limit) [** ] A start job is running for Configur…only root support (16s / no limit) [*** ] A start job is running for Configur…only root support (17s / no limit) [ *** ] A start job is running for Configur…only root support (18s / no limit) [ *** ] A start job is running for Configur…only root support (18s / no limit)[ 48.525630] Key type dns_resolver registered [ ***] A start job is running for Configur…only root support (19s / no limit) [ **] A start job is running for Configur…only root support (19s / no limit) [ *] A start job is running for Configur…only root support (20s / no limit) [ **] A start job is running for Configur…only root support (21s / no limit) [ ***] A start job is running for Configur…only root support (21s / no limit)[ 51.070446] NFS: Registering the id_resolver key type [ 51.072324] Key type id_resolver registered [ 51.073669] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (22s / no limit) [ *** ] A start job is running for Configur…only root support (23s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started dnf makecache --timer. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Login Service. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg102-server login: [ 156.802987] libcfs: loading out-of-tree module taints kernel. [ 156.832520] Key type ._llcrypt registered [ 156.835555] Key type .llcrypt registered [ 156.933247] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_hostid [ 176.784497] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 178.528178] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 178.545968] alg: No test for adler32 (adler32-zlib) [ 180.185457] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 181.310952] LNet: Added LNI 192.168.201.102@tcp [8/256/0/180] [ 183.079705] Key type lgssc registered [ 184.547391] Lustre: Echo OBD driver; http://www.lustre.org/ [ 200.162732] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 249.146100] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 265.192918] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 265.266504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 266.719784] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 266.767993] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 266.875791] Lustre: lustre-MDT0000: new disk, initializing [ 267.038032] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 267.058419] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 272.790840] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 287.897966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 288.103774] Lustre: 6507:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 288.164478] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 288.174028] Lustre: Skipped 1 previous similar message [ 288.279319] Lustre: lustre-MDT0001: new disk, initializing [ 288.366200] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 288.413048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 288.418752] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 293.323478] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 297.896890] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 308.240211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 308.649602] Lustre: lustre-OST0000: new disk, initializing [ 308.653744] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 308.663621] Lustre: 8445:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 308.786685] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 308.955581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 308.968638] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 309.037201] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 315.561155] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 331.511520] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 331.705684] Lustre: lustre-OST0001: new disk, initializing [ 331.709200] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 331.713912] Lustre: 9518:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 331.818171] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 339.014047] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 341.020102] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 341.041144] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 341.145621] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 351.975506] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 365.536864] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 372.709739] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing check_logdir /tmp/testlogs/ [ 379.734424] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing yml_node [ 385.282292] Lustre: DEBUG MARKER: Client: 2.17.54.95 [ 388.565626] Lustre: DEBUG MARKER: MDS: 2.17.54.95 [ 392.202769] Lustre: DEBUG MARKER: OSS: 2.17.54.95 [ 394.184201] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Jun 26 09:32:51 EDT 2026 [ 416.574902] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 418.459367] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 436.704770] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 436.718404] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 436.728348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 438.240675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 438.246330] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 438.259872] Lustre: Skipped 1 previous similar message [ 438.275459] Lustre: Skipped 1 previous similar message [ 443.921482] Lustre: server umount lustre-MDT0000 complete [ 448.487792] LustreError: 6513:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 448.527763] LustreError: 6513:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 448.619477] LustreError: 7455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782480827 with bad export cookie 17734577209378000738 [ 448.621032] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 448.627370] LustreError: 7455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 453.600441] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 453.601275] LustreError: 8964:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 453.601675] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 453.601679] Lustre: Skipped 2 previous similar messages [ 453.626226] Lustre: Skipped 2 previous similar messages [ 453.701121] LustreError: 8964:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 455.128270] Lustre: server umount lustre-MDT0001 complete [ 460.270186] Lustre: server umount lustre-OST0000 complete [ 464.784353] Lustre: server umount lustre-OST0001 complete [ 473.372231] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_hostid [ 486.200623] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 538.039840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 538.433205] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 538.499374] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 538.588695] Lustre: lustre-MDT0000: new disk, initializing [ 538.660360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 538.682972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 544.837145] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 557.286869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 557.435521] Lustre: 15077:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 557.489497] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 557.493683] Lustre: Skipped 1 previous similar message [ 557.565540] Lustre: lustre-MDT0001: new disk, initializing [ 557.677861] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 557.725251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 557.744938] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 564.004408] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 577.177871] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 585.192060] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 594.452570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 595.025717] Lustre: lustre-OST0000: new disk, initializing [ 595.031308] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 595.046429] Lustre: 17049:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 595.219348] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 596.867671] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 596.898768] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 597.018245] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 604.423652] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 616.562677] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 622.393607] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 622.719265] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 628.422392] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 628.765356] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 632.804107] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 632.808979] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 632.832947] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 632.832961] Lustre: Skipped 1 previous similar message [ 637.485535] Lustre: server umount lustre-OST0000 complete [ 643.044897] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 643.048445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 643.058858] Lustre: Skipped 2 previous similar messages [ 643.065389] Lustre: Skipped 2 previous similar messages [ 652.261692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 652.268156] Lustre: Skipped 3 previous similar messages [ 656.351349] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 656.788730] Lustre: server umount lustre-MDT0000 complete [ 657.377202] LustreError: 16303:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 657.396835] LustreError: 16303:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 661.425630] LustreError: 15069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782481040 with bad export cookie 17734577209378002271 [ 661.443264] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 662.267446] Lustre: server umount lustre-MDT0001 complete [ 671.422669] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 09:37:28 (1782481048) [ 698.205880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 699.010037] LustreError: 19929:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 699.183223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 704.487540] LustreError: 19930:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 704.802695] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 714.725367] LustreError: 19929:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 714.751814] LustreError: 19929:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 715.413515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 721.067876] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 730.434580] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 736.539434] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 739.067747] Lustre: Failing over lustre-MDT0000 [ 739.242615] Lustre: server umount lustre-MDT0000 complete [ 740.140814] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 740.151905] LustreError: 19924:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 740.159099] Lustre: Skipped 1 previous similar message [ 740.192610] LustreError: 19924:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 756.707390] Lustre: 3638:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782481119/real 1782481119] req@ffff8adcc6832a00 x1869066335859840/t0(0) o400->MGC192.168.201.102@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1782481135 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 756.734865] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 760.615057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 768.402760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 768.412860] Lustre: Skipped 1 previous similar message [ 768.510642] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 773.601271] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 773.609617] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 773.641790] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 774.419807] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 784.998225] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 791.013640] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 791.023689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 791.050291] Lustre: Skipped 1 previous similar message [ 791.070190] Lustre: Skipped 1 previous similar message [ 804.319454] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 804.699320] Lustre: server umount lustre-MDT0000 complete [ 806.370447] LustreError: 19929:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 806.406206] LustreError: 19929:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 17 previous similar messages [ 808.966622] LustreError: 19910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782481187 with bad export cookie 17734577209378003685 [ 808.969485] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 808.991329] LustreError: 19910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 809.372800] Lustre: server umount lustre-MDT0001 complete [ 819.014538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 825.242302] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 836.196574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 841.865402] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 849.832878] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 855.505613] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 858.033479] Lustre: Failing over lustre-MDT0000 [ 858.051255] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 858.060940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 858.069741] Lustre: Skipped 1 previous similar message [ 858.085554] Lustre: Skipped 4 previous similar messages [ 858.419942] Lustre: server umount lustre-MDT0000 complete [ 871.402845] LustreError: 23271:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 871.449892] LustreError: 23271:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 877.537030] Lustre: 3641:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782481240/real 1782481240] req@ffff8adcc7e15c00 x1869066335892352/t0(0) o400->MGC192.168.201.102@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1782481256 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 877.575979] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 878.690464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 889.107406] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 893.697258] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 894.439963] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 894.448461] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 894.469766] Lustre: Skipped 1 previous similar message [ 894.511719] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 903.668760] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 913.904273] Lustre: server umount lustre-MDT0000 complete [ 918.007894] LustreError: 23254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782481296 with bad export cookie 17734577209378004623 [ 918.020436] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 918.025439] LustreError: 23254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 918.374725] Lustre: server umount lustre-MDT0001 complete [ 927.929124] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 09:41:45 (1782481305) [ 944.707453] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_hostid [ 955.010803] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 994.068552] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 1005.919877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1006.284613] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1006.320933] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1006.413844] Lustre: lustre-MDT0000: new disk, initializing [ 1006.546135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1006.548946] Lustre: Skipped 3 previous similar messages [ 1006.567627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1011.684845] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 1022.529113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1022.633631] Lustre: 29648:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1022.891096] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1027.115858] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 1036.471042] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1041.231751] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1048.162264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1048.409945] Lustre: 31618:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1050.455458] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1050.520480] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 1050.575993] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1050.603767] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 1054.080763] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 1062.951377] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1070.498282] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 1091.553784] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1091.563514] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1091.569452] Lustre: Skipped 3 previous similar messages [ 1091.570695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1091.577129] Lustre: Skipped 6 previous similar messages [ 1093.124903] Lustre: server umount lustre-OST0000 complete [ 1099.744971] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1099.759661] LustreError: Skipped 1 previous similar message [ 1103.373583] Lustre: server umount lustre-MDT0000 complete [ 1106.921921] LustreError: 32339:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1106.939405] LustreError: 32339:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [ 1107.008589] LustreError: 29640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782481485 with bad export cookie 17734577209378005029 [ 1107.015745] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1107.022076] LustreError: 29640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1107.518762] Lustre: server umount lustre-MDT0001 complete [ 1117.979312] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 1121.482383] Key type lgssc unregistered [ 1121.817734] LNet: 33597:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1121.828305] LNetError: 33597:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1121.853438] LNet: Removed LNI 192.168.201.102@tcp [ 1122.894258] Key type .llcrypt unregistered [ 1122.897493] Key type ._llcrypt unregistered [ 1144.318453] Key type ._llcrypt registered [ 1144.321385] Key type .llcrypt registered [ 1144.436370] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 1145.599665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1145.622140] alg: No test for adler32 (adler32-zlib) [ 1146.689764] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 1146.895776] LNet: Added LNI 192.168.201.102@tcp [8/256/0/180] [ 1148.535209] Key type lgssc registered [ 1149.575889] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1155.722616] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1155.732982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1157.123426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1161.739517] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 1167.332899] LustreError: 34991:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1167.350081] LustreError: 34991:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1169.569446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1170.049926] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1174.631789] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 1181.863619] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1186.780360] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1192.505321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1192.786715] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1194.882430] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 1198.475570] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 1205.099370] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1218.840614] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 09:46:36 (1782481596) [ 1220.778290] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1222.400698] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1228.522712] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 09:46:46 (1782481606) [ 1236.326591] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 09:46:54 (1782481614) [ 1237.169199] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1244.696209] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1244.702495] Lustre: Skipped 2 previous similar messages [ 1246.141110] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 09:47:04 (1782481624) [ 1249.010398] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1249.013030] Lustre: Skipped 3 previous similar messages [ 1264.846132] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1264.854499] Lustre: Skipped 6 previous similar messages [ 1267.074499] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 09:47:25 (1782481645) [ 1269.824389] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1272.777428] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1280.714539] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 09:47:38 (1782481658) [ 1282.336758] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1282.344329] Lustre: Skipped 3 previous similar messages [ 1293.828584] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 09:47:51 (1782481671) [ 1307.616142] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1307.623323] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1307.642388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1310.692193] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1310.699355] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1310.703619] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1310.726321] Lustre: Skipped 1 previous similar message [ 1313.628399] Lustre: server umount lustre-OST0000 complete [ 1318.368986] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1318.375926] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1318.388809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1320.931081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1320.943405] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1323.576394] Lustre: server umount lustre-MDT0000 complete [ 1326.054574] LustreError: 34991:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1326.066468] LustreError: 34991:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1327.118688] LustreError: 38547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782481706 with bad export cookie 11885550398137331139 [ 1327.122265] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1327.126473] LustreError: 38547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1327.353585] Lustre: server umount lustre-MDT0001 complete [ 1336.417581] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 1338.381279] Key type lgssc unregistered [ 1338.568739] LNet: 42841:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1338.575628] LNetError: 42841:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1338.586827] LNet: Removed LNI 192.168.201.102@tcp [ 1339.207677] Key type .llcrypt unregistered [ 1339.209718] Key type ._llcrypt unregistered [ 1353.313785] Key type ._llcrypt registered [ 1353.318517] Key type .llcrypt registered [ 1353.387405] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_hostid [ 1368.006988] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 1368.925280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1369.016680] alg: No test for adler32 (adler32-zlib) [ 1369.982916] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 1370.197838] LNet: Added LNI 192.168.201.102@tcp [8/256/0/180] [ 1371.903148] Key type lgssc registered [ 1372.857907] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1400.860973] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1400.877305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1402.084740] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1402.104870] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1402.147395] Lustre: lustre-MDT0000: new disk, initializing [ 1402.192330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1402.212159] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1405.413458] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 1413.743227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1413.844600] Lustre: 45997:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1413.888450] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1413.891183] Lustre: Skipped 1 previous similar message [ 1413.932742] Lustre: lustre-MDT0001: new disk, initializing [ 1413.981828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1414.013428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1414.021793] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1417.227757] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 1424.389252] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1428.116709] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1433.805553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1434.063791] Lustre: lustre-OST0000: new disk, initializing [ 1434.066477] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1434.069669] Lustre: 47962:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1434.156842] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1439.061709] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 1441.814222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1441.821102] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1441.893104] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1446.600549] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1451.153631] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1451.316555] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1454.487094] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1454.631731] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1457.121394] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1457.123484] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1457.131980] LustreError: Skipped 1 previous similar message [ 1457.132865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1457.137993] Lustre: Skipped 1 previous similar message [ 1462.322026] Lustre: server umount lustre-OST0000 complete [ 1465.314307] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1465.319188] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1465.330599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1465.334260] Lustre: Skipped 1 previous similar message [ 1467.361340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1467.362465] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1472.490472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1472.500866] Lustre: Skipped 2 previous similar messages [ 1477.600965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1477.604665] Lustre: Skipped 1 previous similar message [ 1479.135183] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1479.470275] Lustre: server umount lustre-MDT0000 complete [ 1482.725122] LustreError: 47127:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1482.746000] LustreError: 47127:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1482.890136] LustreError: 45989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782481861 with bad export cookie 15933027532012803936 [ 1482.897495] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1482.910685] LustreError: 45989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1483.231940] Lustre: server umount lustre-MDT0001 complete [ 1491.565398] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 09:51:09 (1782481869) [ 1497.450679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1498.074417] LustreError: 50423:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1498.145303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1501.595686] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 1503.203689] LustreError: 50424:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1508.320151] LustreError: 50423:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1508.553953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1508.840906] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1512.748982] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 1520.233243] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1525.376294] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1532.035481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1532.437300] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1538.441342] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 1545.395246] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1550.902530] LustreError: 52950:0:(llog.c:444:llog_init_handle()) llog type is not specified! [ 1551.740152] Lustre: Modifying parameter general.timeout=1 in log params [ 1553.386810] Lustre: Modifying parameter general.timeout=3 in log params [ 1553.390757] Lustre: Skipped 1 previous similar message [ 1555.629536] Lustre: Modifying parameter general.timeout=6 in log params [ 1555.633694] Lustre: Skipped 2 previous similar messages [ 1559.980450] Lustre: Modifying parameter general.timeout=12 in log params [ 1559.985132] Lustre: Skipped 5 previous similar messages [ 1568.694328] Lustre: Modifying parameter general.timeout=24 in log params [ 1568.698883] Lustre: Skipped 11 previous similar messages [ 1584.714482] Lustre: Modifying parameter general.timeout=45 in log params [ 1584.718142] Lustre: Skipped 20 previous similar messages [ 1616.759219] Lustre: Modifying parameter general.timeout=88 in log params [ 1616.763694] Lustre: Skipped 42 previous similar messages [ 1655.232909] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 09:53:53 (1782482033) [ 1657.898114] Lustre: 59878:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1659.190872] Lustre: 59976:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1661.554889] Lustre: 60172:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1661.561397] Lustre: 60172:0:(mgs_llog.c:1447:mgs_modify_param()) Skipped 1 previous similar message [ 1684.158319] Lustre: 61749:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1684.170720] Lustre: 61749:0:(mgs_llog.c:1447:mgs_modify_param()) Skipped 2 previous similar messages [ 1684.183936] LustreError: 61749:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1684.910470] Lustre: Disabling parameter general.ost.OSS.ost_io.nrs_policies= in log params [ 1684.914265] Lustre: Skipped 62 previous similar messages [ 1687.535308] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 09:54:25 (1782482065) [ 1698.274756] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1698.280428] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1698.289532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1702.149932] Lustre: server umount lustre-MDT0000 complete [ 1704.599552] LustreError: 50403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782482083 with bad export cookie 15933027532012804818 [ 1704.601709] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1704.609168] LustreError: 50403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 1704.749447] Lustre: server umount lustre-MDT0001 complete [ 1718.012993] Lustre: server umount lustre-OST0000 complete [ 1739.653669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1739.928696] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1740.125347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1743.496569] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 1750.170714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1750.245712] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1750.278652] Lustre: 64280:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1754.405864] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 1760.158173] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1764.352453] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1769.584705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1769.794832] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1774.481675] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 1779.986527] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1781.299740] Lustre: 66845:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1787.873480] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1787.880948] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1787.893286] Lustre: Skipped 2 previous similar messages [ 1787.896952] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1787.900960] Lustre: Skipped 3 previous similar messages [ 1792.058433] Lustre: server umount lustre-OST0000 complete [ 1796.577021] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1796.584900] LustreError: Skipped 1 previous similar message [ 1796.587281] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1796.600428] Lustre: Skipped 1 previous similar message [ 1800.869842] Lustre: server umount lustre-MDT0000 complete [ 1803.643843] LustreError: 65021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782482182 with bad export cookie 15933027532012807646 [ 1803.645264] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1803.650888] LustreError: 65021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1803.904531] Lustre: server umount lustre-MDT0001 complete [ 1812.077157] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 1814.035785] Key type lgssc unregistered [ 1814.287937] LNet: 68053:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1814.293460] LNetError: 68053:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1814.308369] LNet: Removed LNI 192.168.201.102@tcp [ 1814.936657] Key type .llcrypt unregistered [ 1814.938945] Key type ._llcrypt unregistered [ 1832.572694] Key type ._llcrypt registered [ 1832.573880] Key type .llcrypt registered [ 1832.629256] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 1833.469448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1833.542137] alg: No test for adler32 (adler32-zlib) [ 1834.541850] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 1834.757200] LNet: Added LNI 192.168.201.102@tcp [8/256/0/180] [ 1836.400295] Key type lgssc registered [ 1837.318707] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1843.555376] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1843.569155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1844.950770] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1848.969556] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 1854.437124] LustreError: 69445:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1854.444269] LustreError: 69445:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1855.942542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1856.427048] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1860.036433] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 1865.531720] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1870.348583] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1876.531701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1877.024740] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1882.811412] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 1888.657228] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1896.036978] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1902.564244] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1902.569443] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1902.580882] LustreError: 69445:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1904.200817] Lustre: server umount lustre-MDT0000 complete [ 1904.609783] LustreError: 69446:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1904.614176] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1907.024086] LustreError: 69425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782482285 with bad export cookie 4003778978844822629 [ 1907.038147] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1907.038430] LustreError: 69425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1907.281757] Lustre: server umount lustre-MDT0001 complete [ 1921.206310] Lustre: server umount lustre-OST0000 complete [ 1943.428409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1943.667540] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1943.691728] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1943.964135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1948.253929] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 1955.498900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1955.561647] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1955.582949] Lustre: 74310:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1955.923640] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1958.799894] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 1964.402028] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1969.088578] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1974.523532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1974.663856] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1974.936247] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1979.222382] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 1985.127864] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1986.476491] Lustre: 76885:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1987.929928] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1993.186348] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1993.193363] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1993.197645] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1993.217165] Lustre: Skipped 2 previous similar messages [ 1996.877891] Lustre: server umount lustre-OST0000 complete [ 2001.890317] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2001.896310] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2001.908423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2001.917019] Lustre: Skipped 1 previous similar message [ 2003.425126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2008.549094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2008.554480] Lustre: Skipped 2 previous similar messages [ 2013.666877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2013.673860] Lustre: Skipped 1 previous similar message [ 2014.175861] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2014.315983] Lustre: server umount lustre-MDT0000 complete [ 2017.141903] LustreError: 75069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782482396 with bad export cookie 4003778978844823462 [ 2017.146600] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2017.152875] LustreError: 75069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2017.404847] Lustre: server umount lustre-MDT0001 complete [ 2025.190717] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 2027.223768] Key type lgssc unregistered [ 2027.421584] LNet: 78097:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2027.429028] LNetError: 78097:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2027.449523] LNet: Removed LNI 192.168.201.102@tcp [ 2028.009457] Key type .llcrypt unregistered [ 2028.012527] Key type ._llcrypt unregistered [ 2045.718439] Key type ._llcrypt registered [ 2045.723035] Key type .llcrypt registered [ 2045.812194] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 2046.562604] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2046.597135] alg: No test for adler32 (adler32-zlib) [ 2047.566722] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 2047.766570] LNet: Added LNI 192.168.201.102@tcp [8/256/0/180] [ 2049.407187] Key type lgssc registered [ 2050.198440] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2056.321457] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2056.339414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2057.807974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2061.519461] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 2067.939250] LustreError: 79493:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2067.954839] LustreError: 79493:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2068.128613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2068.541867] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2072.146435] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 2077.988154] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2082.230413] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2087.423227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2087.819759] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2092.449799] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 2097.846682] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2107.824835] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2108.613594] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2109.827679] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2109.833226] Lustre: Skipped 1 previous similar message [ 2112.582357] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2112.585168] Lustre: Skipped 3 previous similar messages [ 2116.671400] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2116.674852] Lustre: Skipped 5 previous similar messages [ 2125.212314] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2125.220236] Lustre: Skipped 11 previous similar messages [ 2141.350512] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2141.354710] Lustre: Skipped 23 previous similar messages [ 2173.481577] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2173.484978] Lustre: Skipped 44 previous similar messages [ 2184.366598] Lustre: 87588:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2186.274604] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 10:02:44 (1782482564) [ 2187.410129] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2188.692987] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 10:02:46 (1782482566) [ 2194.400394] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2194.406576] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2194.409148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2194.419386] Lustre: Skipped 1 previous similar message [ 2197.649336] Lustre: server umount lustre-OST0000 complete [ 2201.568801] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2201.581218] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2201.593857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2201.599839] Lustre: Skipped 1 previous similar message [ 2204.642813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2204.643588] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2206.342821] Lustre: server umount lustre-MDT0000 complete [ 2208.977587] LustreError: 79473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782482587 with bad export cookie 17241776544333799786 [ 2208.980137] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2208.989861] LustreError: 79473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2209.259324] Lustre: server umount lustre-MDT0001 complete [ 2217.425526] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 2218.906566] Key type lgssc unregistered [ 2219.098629] LNet: 88836:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2219.104637] LNetError: 88836:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2219.120185] LNet: Removed LNI 192.168.201.102@tcp [ 2219.674779] Key type .llcrypt unregistered [ 2219.676874] Key type ._llcrypt unregistered [ 2222.188477] Key type ._llcrypt registered [ 2222.189623] Key type .llcrypt registered [ 2222.270989] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2225.120599] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules [ 2225.879237] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2225.891163] alg: No test for adler32 (adler32-zlib) [ 2226.930387] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 2227.107332] LNet: Added LNI 192.168.201.102@tcp [8/256/0/180] [ 2228.736107] Key type lgssc registered [ 2229.563901] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2230.475196] LustreError: 90037:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2242.483246] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 2248.361556] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2252.352186] LustreError: 90037:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2252.392385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2253.778960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2256.608902] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 2262.262430] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 10:04:00 (1782482640) [ 2264.036223] LustreError: 91060:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2264.044045] LustreError: 91060:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2265.292820] LustreError: 91092:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2265.305077] LustreError: 91092:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2265.311368] LustreError: 91092:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 12, retries 0, failed: rc = -5 [ 2265.585658] Lustre: server umount lustre-MDT0000 complete [ 2274.459306] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 2276.166872] Key type lgssc unregistered [ 2276.376722] LNet: 92583:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2276.382743] LNetError: 92583:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2276.401912] LNet: Removed LNI 192.168.201.102@tcp [ 2276.929761] Key type .llcrypt unregistered [ 2276.932335] Key type ._llcrypt unregistered [ 2288.056812] Key type ._llcrypt registered [ 2288.058390] Key type .llcrypt registered [ 2288.113533] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 2288.794070] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2288.814528] alg: No test for adler32 (adler32-zlib) [ 2289.729530] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 2289.834947] LNet: Added LNI 192.168.201.102@tcp [8/256/0/180] [ 2291.447266] Key type lgssc registered [ 2292.169953] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2296.782882] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2296.794769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2298.066296] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2300.621352] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 2305.997343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2306.414235] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2308.778746] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 2313.935369] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2317.725350] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2322.408609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2322.800633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2327.382800] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 2333.514737] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2369.164497] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 10:05:47 (1782482747) [ 2370.623649] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2372.027249] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 10:05:50 (1782482750) [ 2377.190159] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2377.195507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2377.196846] Lustre: Skipped 1 previous similar message [ 2377.210878] Lustre: Skipped 2 previous similar messages [ 2380.706859] Lustre: server umount lustre-MDT0000 complete [ 2382.306052] LustreError: 93778:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2382.316177] LustreError: 93778:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2382.934686] LustreError: 93759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782482761 with bad export cookie 14002821123890724091 [ 2382.940792] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2382.945195] LustreError: 93759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2383.145859] Lustre: server umount lustre-MDT0001 complete [ 2396.109622] Lustre: server umount lustre-OST0000 complete [ 2404.275702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2404.595968] LustreError: 98052:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2404.724700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2407.383808] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 2409.952797] LustreError: 98053:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2412.681163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2413.138507] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2416.340823] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 2421.352531] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2424.490487] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2433.613489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2433.715122] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2433.718844] LustreError: 98042:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2433.728552] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2433.732675] LustreError: 100134:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2433.735203] LustreError: 100134:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2433.737818] LustreError: 100134:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2433.767785] Lustre: server umount lustre-OST0000 complete [ 2433.769235] LustreError: 100134:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2439.359950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2439.479674] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2439.490415] LustreError: 98040:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2439.498076] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2439.503235] LustreError: 100595:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2439.509396] LustreError: 100595:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2439.513314] LustreError: 100595:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2439.569364] Lustre: server umount lustre-OST0000 complete [ 2439.571424] LustreError: 100595:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2446.063049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2446.237586] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2446.241501] Lustre: Found index 0 for lustre-OST0000, updating log [ 2446.245553] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2446.316925] Lustre: lustre-OST0000: new disk, initializing [ 2446.323488] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2446.331565] Lustre: 101129:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2446.533291] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2450.526380] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 2453.143845] Lustre: Failing over lustre-OST0000 [ 2453.318265] Lustre: server umount lustre-OST0000 complete [ 2458.018407] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2458.028780] Lustre: Skipped 1 previous similar message [ 2458.034155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2458.038425] Lustre: Skipped 1 previous similar message [ 2463.200538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2463.208501] Lustre: Skipped 1 previous similar message [ 2468.323386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2468.331104] Lustre: Skipped 1 previous similar message [ 2470.367201] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2470.500591] Lustre: server umount lustre-MDT0000 complete [ 2472.958294] LustreError: 98033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782482851 with bad export cookie 14002821123890726821 [ 2472.960942] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2472.965038] LustreError: 98033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2473.149450] Lustre: server umount lustre-MDT0001 complete [ 2479.363859] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 10:07:37 (1782482857) [ 2490.064766] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 2496.106669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2496.306910] LustreError: 103474:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2496.384801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2498.952566] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 2501.601460] LustreError: 103475:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2504.146648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2507.290511] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 2509.150681] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2513.038417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2513.464611] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2514.476231] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2514.479467] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 2514.486891] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2517.430801] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 2519.526679] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 2519.544477] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2521.575717] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:02 [ 2521.588037] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2521.607990] Lustre: lustre-OST0000: Recovery over after 0:07, of 1 clients 1 recovered and 0 were evicted. [ 2521.612428] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2521.629651] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2523.358255] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2523.427760] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2523.479411] Lustre: lustre-OST0001: new disk, initializing [ 2523.481783] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2523.485533] Lustre: 105794:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2527.613323] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 2532.387724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2532.392709] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2532.445337] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2532.948940] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2537.207434] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2539.905321] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.2@tcp (stopping) [ 2542.563955] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2542.566993] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2542.577912] Lustre: Skipped 3 previous similar messages [ 2550.137544] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.2@tcp (stopping) [ 2550.148099] Lustre: Skipped 7 previous similar messages [ 2554.335519] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2554.538582] Lustre: server umount lustre-MDT0001 complete [ 2555.241615] LustreError: 103470:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2560.182196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2560.232948] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2560.237930] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2560.262437] Lustre: 103461:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2560.689943] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2560.692679] Lustre: Skipped 3 previous similar messages [ 2561.460843] LustreError: 107374:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2561.467377] LustreError: 107374:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2561.474986] LustreError: 103466:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2561.523207] LustreError: 107381:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.102@tcp: cfg command failed: rc = -17 [ 2561.528758] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2561.528758] [ 2562.903038] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 2566.131569] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2568.472814] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 10:09:06 (1782482946) [ 2578.417263] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 2588.266140] Lustre: 109401:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2600.604485] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2602.467576] Lustre: 110533:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2662.880806] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2662.886358] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2662.897181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2662.899272] Lustre: Skipped 4 previous similar messages [ 2668.671460] Lustre: server umount lustre-MDT0000 complete [ 2670.552019] LustreError: 105793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782483049 with bad export cookie 14002821123890727745 [ 2670.552835] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2670.557115] LustreError: 105793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2670.751034] Lustre: server umount lustre-MDT0001 complete [ 2682.895967] Lustre: server umount lustre-OST0000 complete [ 2694.251065] Lustre: server umount lustre-OST0001 complete [ 2696.780024] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2701.221912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2704.939543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2709.394101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2719.364243] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 2724.350812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2724.364300] Lustre: lustre-MDT0000: reset Object Index mappings [ 2724.598815] LustreError: 114484:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2724.608455] LustreError: 114484:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2724.657852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2726.432611] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 2730.348371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2730.358084] Lustre: lustre-MDT0001: reset Object Index mappings [ 2732.586061] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 2733.845983] Lustre: 115630:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2736.595895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2739.543535] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 2741.035432] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 2743.893804] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2747.209556] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 2749.358781] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 2749.359450] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 2749.362645] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:381 to 0x280000401:418) [ 2757.046699] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2758.972296] Lustre: 117506:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2784.059252] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 10:12:42 (1782483162) [ 2815.969745] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2815.976338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2815.979699] Lustre: Skipped 5 previous similar messages [ 2815.984084] Lustre: Skipped 9 previous similar messages [ 2819.383972] Lustre: server umount lustre-MDT0000 complete [ 2821.088539] LustreError: 117521:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2821.102900] LustreError: 117521:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2821.337967] LustreError: 115633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782483200 with bad export cookie 14002821123890774771 [ 2821.341638] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2821.345193] LustreError: 115633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2826.210112] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2826.220014] Lustre: Skipped 2 previous similar messages [ 2827.664616] Lustre: server umount lustre-MDT0001 complete [ 2835.813395] Lustre: server umount lustre-OST0000 complete [ 2852.319180] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2852.427273] Lustre: server umount lustre-OST0001 complete [ 2855.944745] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_hostid [ 2861.148974] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 2882.272828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2882.402330] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2882.419648] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2882.500051] Lustre: lustre-MDT0000: new disk, initializing [ 2882.544640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2882.547635] Lustre: Skipped 3 previous similar messages [ 2882.559854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2884.889422] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 2889.178728] Lustre: server umount lustre-MDT0000 complete [ 2894.029707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2894.194524] Lustre: Found index 0 for lustre-MDT0000, updating log [ 2894.205370] Lustre: 121763:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2894.212393] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 2896.377305] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 2898.937511] Lustre: server umount lustre-MDT0000 complete [ 2903.818823] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 10:14:42 (1782483282) [ 2904.685964] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 2905.648676] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 10:14:44 (1782483284) [ 2914.465607] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 2919.934268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2922.673325] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 2927.473443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2927.563345] Lustre: 123812:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2927.601156] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2927.604462] Lustre: Skipped 1 previous similar message [ 2927.656813] Lustre: lustre-MDT0001: new disk, initializing [ 2927.757389] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2927.769930] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2930.059061] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 2933.265956] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2937.476316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2937.656842] Lustre: lustre-OST0000: new disk, initializing [ 2937.659797] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2937.664404] Lustre: 125386:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2939.031532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2939.042240] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2939.096658] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2940.584486] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 2946.412706] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2946.534658] Lustre: lustre-OST0001: new disk, initializing [ 2946.536990] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2946.541229] Lustre: 126260:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2947.812163] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2947.817525] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2947.901064] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2949.900630] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 2956.580325] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2958.616857] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3152.827804] Lustre: 132640:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3152.831682] Lustre: 132640:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3152.834896] Lustre: 132640:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3152.837803] Lustre: 132640:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3152.845060] Lustre: 132640:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3152.851553] Lustre: 132640:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3301.663504] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 10:21:19 (1782483679) [ 3316.709239] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3316.713284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3316.718660] Lustre: Skipped 3 previous similar messages [ 3316.723562] Lustre: Skipped 6 previous similar messages [ 3320.176644] Lustre: server umount lustre-MDT0000 complete [ 3321.825766] LustreError: 127072:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3321.832735] LustreError: 127072:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 3322.117557] LustreError: 132670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782483701 with bad export cookie 14002821123890834341 [ 3322.126485] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3322.126493] LustreError: 132670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3322.362871] Lustre: server umount lustre-MDT0001 complete [ 3340.244821] Lustre: server umount lustre-OST0000 complete [ 3358.569878] Lustre: server umount lustre-OST0001 complete [ 3361.384170] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_hostid [ 3366.418682] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 3385.081593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3385.217177] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3385.238445] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3385.284366] Lustre: lustre-MDT0000: new disk, initializing [ 3385.322943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3385.331975] Lustre: Skipped 5 previous similar messages [ 3385.341426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3387.775601] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 3393.552793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3393.602816] Lustre: 135668:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3393.769741] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3396.041758] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 3401.227936] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3403.838612] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3407.104070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3407.247043] Lustre: 137637:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3408.517213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3408.520661] Lustre: Skipped 1 previous similar message [ 3408.523299] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3408.563379] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3410.206308] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 3415.265358] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3419.793696] Lustre: lustre-MDD0000: changelog on [ 3421.302761] Lustre: lustre-MDD0001: changelog on [ 4463.005907] Lustre: lustre-MDD0001: changelog off [ 4463.902886] Lustre: 138659:0:(llog_cat.c:977:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 4464.384070] Lustre: lustre-MDD0000: changelog off [ 4468.706510] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4468.709197] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4468.720216] LustreError: Skipped 1 previous similar message [ 4468.731220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4468.735798] Lustre: Skipped 1 previous similar message [ 4473.156364] Lustre: server umount lustre-OST0000 complete [ 4478.948364] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4478.957193] Lustre: Skipped 2 previous similar messages [ 4481.187838] Lustre: server umount lustre-MDT0000 complete [ 4483.119604] LustreError: 135660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782484862 with bad export cookie 14002821123891643975 [ 4483.120508] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4483.126053] LustreError: 135660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4483.306180] Lustre: server umount lustre-MDT0001 complete [ 4488.954136] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 4490.494043] Key type lgssc unregistered [ 4490.700677] LNet: 146524:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4490.705130] LNetError: 146524:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4490.725803] LNet: Removed LNI 192.168.201.102@tcp [ 4491.201173] Key type .llcrypt unregistered [ 4491.203812] Key type ._llcrypt unregistered [ 4500.336800] Key type ._llcrypt registered [ 4500.338296] Key type .llcrypt registered [ 4500.385024] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_hostid [ 4507.427717] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 4507.783787] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4507.873090] alg: No test for adler32 (adler32-zlib) [ 4508.786564] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 4508.905459] LNet: Added LNI 192.168.201.102@tcp [8/256/0/180] [ 4510.527375] Key type lgssc registered [ 4511.028910] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4545.193605] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4545.226051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4546.558155] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4546.581355] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4546.648090] Lustre: lustre-MDT0000: new disk, initializing [ 4546.759985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4546.800667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4550.971430] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 4566.356618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4566.757509] Lustre: 149703:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4566.893428] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4566.913835] Lustre: Skipped 1 previous similar message [ 4567.104338] Lustre: lustre-MDT0001: new disk, initializing [ 4567.302660] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4567.386589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4567.401490] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4575.328216] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 4589.940944] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4599.979779] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4608.990616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4609.300126] Lustre: lustre-OST0000: new disk, initializing [ 4609.305376] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4609.311766] Lustre: 151673:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4609.410045] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4616.470993] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 4618.824990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4618.834218] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4618.940595] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4625.754380] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4632.108471] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_module obdecho/obdecho [ 4634.291454] LustreError: 152677:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 4634.299665] LustreError: 152677:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 4635.808887] LustreError: 152773:0:(echo_client.c:1883:echo_md_handler()) Can not resolve the path : rc = -2 [ 4649.440538] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4649.445688] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4649.457515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4649.959275] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4649.968173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4649.980973] Lustre: Skipped 1 previous similar message [ 4652.636742] Lustre: server umount lustre-MDT0000 complete [ 4655.932849] LustreError: 149694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782485034 with bad export cookie 13092516133881356155 [ 4655.946237] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4655.958905] LustreError: 149694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4656.220937] Lustre: server umount lustre-MDT0001 complete [ 4669.424249] Lustre: server umount lustre-OST0000 complete [ 4688.995634] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_hostid [ 4698.232175] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 4729.034934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4729.301640] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4729.352602] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4729.457851] Lustre: lustre-MDT0000: new disk, initializing [ 4729.548163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4729.575175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4732.874165] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 4742.637018] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4742.777458] Lustre: 156489:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4742.849368] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4742.853885] Lustre: Skipped 1 previous similar message [ 4742.929363] Lustre: lustre-MDT0001: new disk, initializing [ 4743.078525] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4743.145097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4743.154927] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4747.305959] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 4755.260742] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4759.368056] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4764.588267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4764.822425] Lustre: lustre-OST0000: new disk, initializing [ 4764.825319] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4764.829674] Lustre: 158453:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4764.893242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4769.398250] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 4775.005942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4775.019249] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4775.094896] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4778.408571] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4783.346362] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4783.513956] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4788.068245] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4788.345858] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4790.752309] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4790.753214] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4790.764127] LustreError: Skipped 1 previous similar message [ 4790.768061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4790.783695] Lustre: Skipped 2 previous similar messages [ 4795.874671] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4795.881077] Lustre: Skipped 2 previous similar messages [ 4796.611808] Lustre: server umount lustre-OST0000 complete [ 4800.993920] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4801.006114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4801.009917] Lustre: Skipped 1 previous similar message [ 4806.118847] LustreError: 156500:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4806.140742] LustreError: 156500:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4807.942373] Lustre: server umount lustre-MDT0000 complete [ 4811.237355] LustreError: 156500:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4811.245786] LustreError: 156500:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4811.516400] LustreError: 157282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782485190 with bad export cookie 13092516133881357107 [ 4811.523197] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4811.531841] LustreError: 157282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4811.867991] Lustre: server umount lustre-MDT0001 complete [ 4815.497452] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 10:46:33 (1782485193) [ 4816.890655] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 4818.520358] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 10:46:36 (1782485196) [ 4825.407386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4825.815589] LustreError: 160866:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4825.883593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4829.782801] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 4831.200342] LustreError: 160867:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4835.312914] LustreError: 160866:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4837.958492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4842.341970] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 4849.950686] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4855.189650] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4862.411879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4862.842663] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4862.849328] Lustre: Skipped 1 previous similar message [ 4868.890238] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 4876.053851] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4882.108822] Lustre: server umount lustre-MDT0000 complete [ 4884.460883] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4884.468677] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4884.482343] Lustre: Skipped 1 previous similar message [ 4884.487368] LustreError: 161910:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4885.681347] LustreError: 160847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782485264 with bad export cookie 13092516133881357996 [ 4885.691658] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4886.501090] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4892.106635] Lustre: server umount lustre-MDT0001 complete [ 4898.485129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4910.763743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4911.185714] LustreError: 164707:0:(llog.c:1646:llog_backup()) MGC192.168.201.102@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 4911.199493] Lustre: 164707:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.102@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 4915.184620] LustreError: 164713:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4915.202190] LustreError: 164713:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4948.333560] LustreError: 164712:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4948.347448] LustreError: 164712:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 30 previous similar messages [ 4972.512146] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 4972.521799] Lustre: 164707:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.201.102@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 5013.501124] LustreError: 164712:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5013.507453] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5013.509708] LustreError: 164712:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 41 previous similar messages [ 5016.998031] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 5018.615168] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5023.845303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5031.904350] LustreError: 147925:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000004ecfc452 x1869070872999168/t0(0) o250->MGC192.168.201.102@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5035.352157] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 5037.543977] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5040.472743] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5041.533790] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5045.156969] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5046.250652] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5062.313074] Lustre: server umount lustre-MDT0000 complete [ 5063.138688] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5063.145296] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5063.148479] Lustre: Skipped 3 previous similar messages [ 5065.044190] LustreError: 160847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782485443 with bad export cookie 13092516133881364450 [ 5065.048226] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5065.052924] LustreError: 160847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 5065.259740] Lustre: server umount lustre-MDT0001 complete [ 5078.426789] Lustre: server umount lustre-OST0000 complete [ 5095.112621] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_hostid [ 5102.560489] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 5130.801857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5131.030814] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5131.065100] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5131.121829] Lustre: lustre-MDT0000: new disk, initializing [ 5131.218210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5134.803235] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 5143.797046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5143.874628] Lustre: 169812:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5143.989163] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5143.994152] Lustre: Skipped 2 previous similar messages [ 5144.019626] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5147.525788] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 5154.919709] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5159.048093] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5164.432313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5164.669166] Lustre: lustre-OST0000: new disk, initializing [ 5164.671287] Lustre: Skipped 1 previous similar message [ 5164.673853] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5164.676267] Lustre: Skipped 2 previous similar messages [ 5164.679412] Lustre: 171779:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5169.666477] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 5171.239229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5171.246427] Lustre: Skipped 1 previous similar message [ 5171.249128] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5171.327744] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5176.976897] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5180.691669] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5180.884862] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5184.649691] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5184.836640] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5186.529498] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5186.541142] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5186.548217] Lustre: Skipped 1 previous similar message [ 5186.553158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5186.556241] Lustre: Skipped 2 previous similar messages [ 5192.407483] Lustre: server umount lustre-OST0000 complete [ 5195.241061] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5195.247096] LustreError: Skipped 1 previous similar message [ 5201.314085] Lustre: server umount lustre-MDT0000 complete [ 5202.400785] LustreError: 172468:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5202.428307] LustreError: 172468:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 20 previous similar messages [ 5204.543847] LustreError: 169806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782485583 with bad export cookie 13092516133881381257 [ 5204.552859] LustreError: 169806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5204.554638] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5204.944577] Lustre: server umount lustre-MDT0001 complete [ 5211.874447] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 10:53:09 (1782485589) [ 5216.103741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5219.083688] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 5224.427619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5227.312460] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 5231.676861] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5235.181515] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5239.640892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5243.062442] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 5247.386666] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5257.646454] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 10:53:55 (1782485635) [ 5260.260603] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5260.260781] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5260.269959] Lustre: Skipped 3 previous similar messages [ 5260.271444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5260.290883] Lustre: Skipped 8 previous similar messages [ 5266.257132] Lustre: server umount lustre-OST0000 complete [ 5274.565833] Lustre: server umount lustre-MDT0000 complete [ 5276.957647] LustreError: 174171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782485655 with bad export cookie 13092516133881382139 [ 5276.962420] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5276.963808] LustreError: 174171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5277.209756] Lustre: server umount lustre-MDT0001 complete [ 5284.276471] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 5286.107833] Key type lgssc unregistered [ 5286.294854] LNet: 177911:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5286.300661] LNetError: 177911:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5286.312754] LNet: Removed LNI 192.168.201.102@tcp [ 5286.818524] Key type .llcrypt unregistered [ 5286.820565] Key type ._llcrypt unregistered [ 5298.697217] Key type ._llcrypt registered [ 5298.698895] Key type .llcrypt registered [ 5298.767926] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 5299.417207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5299.527310] alg: No test for adler32 (adler32-zlib) [ 5300.509313] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 5300.648416] LNet: Added LNI 192.168.201.102@tcp [8/256/0/180] [ 5302.255242] Key type lgssc registered [ 5303.459359] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5310.204877] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5310.227379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5325.727478] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5330.854204] LustreError: 179148:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.102@tcp: failed processing log, type 4: rc = -110 [ 5336.032305] LustreError: 179148:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 5336.071220] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 5336.093526] LustreError: MGC192.168.201.102@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 5336.121286] LustreError: 179148:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 5336.162724] LustreError: 179148:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 5336.188830] LustreError: 179148:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 5336.351996] Lustre: server umount lustre-OST0000 complete [ 5336.373682] LustreError: 179148:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5350.744647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5352.467624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5359.121326] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 5362.656591] LustreError: 179650:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5362.676174] LustreError: 179650:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5363.689551] LustreError: 179651:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5368.805024] LustreError: 179650:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5371.514630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5372.141714] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5379.334560] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 5391.473970] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5400.236642] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5408.032402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5408.235338] LustreError: 181470:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 5408.239981] Lustre: 181470:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.201.102@tcp: invalid local config log lustre-OST0000: rc = -22 [ 5408.402066] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5415.421622] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 5424.098939] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5428.708481] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5428.713945] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5428.729057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5430.241981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5430.243801] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5430.248974] Lustre: Skipped 1 previous similar message [ 5430.291326] Lustre: Skipped 1 previous similar message [ 5433.147691] Lustre: server umount lustre-MDT0000 complete [ 5435.372020] LustreError: 179650:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5435.389616] LustreError: 179650:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5436.784285] LustreError: 179631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782485815 with bad export cookie 14583298288864474167 [ 5436.807903] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5437.193375] Lustre: server umount lustre-MDT0001 complete [ 5450.535699] Lustre: server umount lustre-OST0000 complete [ 5460.684871] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_hostid [ 5472.301860] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 5514.761121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5515.031680] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5515.062867] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5515.123331] Lustre: lustre-MDT0000: new disk, initializing [ 5515.208977] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5515.223314] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5520.552534] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 5533.135351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5533.256507] Lustre: 184998:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5533.341946] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5533.348438] Lustre: Skipped 1 previous similar message [ 5533.407195] Lustre: lustre-MDT0001: new disk, initializing [ 5533.490330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5533.526934] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5533.545070] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5538.312630] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 5548.743911] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5554.975872] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5562.519830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5562.806740] Lustre: lustre-OST0000: new disk, initializing [ 5562.811929] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5562.819663] Lustre: 186964:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5562.897341] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5568.348396] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 5572.167268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5572.181342] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5572.259314] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5577.310773] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5582.112632] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5582.277024] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5588.079586] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5588.332974] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5593.062126] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5593.062732] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5593.069357] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5593.069372] Lustre: Skipped 1 previous similar message [ 5593.069695] LustreError: Skipped 1 previous similar message [ 5593.112125] Lustre: Skipped 1 previous similar message [ 5596.922478] Lustre: server umount lustre-OST0000 complete [ 5603.299768] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5603.324406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5603.338894] Lustre: Skipped 2 previous similar messages [ 5608.419797] LustreError: 185008:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5608.436869] LustreError: 185008:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5608.655078] Lustre: server umount lustre-MDT0000 complete [ 5613.048152] LustreError: 184988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782485991 with bad export cookie 14583298288864474888 [ 5613.051685] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5613.061725] LustreError: 184988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5613.582228] Lustre: server umount lustre-MDT0001 complete [ 5623.761717] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 11:00:01 (1782486001) [ 5646.938096] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 5650.740513] Key type lgssc unregistered [ 5651.139656] LNet: 189909:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5651.144732] LNetError: 189909:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5651.168936] LNet: Removed LNI 192.168.201.102@tcp [ 5652.529701] Key type .llcrypt unregistered [ 5652.532705] Key type ._llcrypt unregistered [ 5674.987362] Key type ._llcrypt registered [ 5674.990410] Key type .llcrypt registered [ 5675.129882] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 5676.547564] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5676.610577] alg: No test for adler32 (adler32-zlib) [ 5677.637684] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 5677.843388] LNet: Added LNI 192.168.201.102@tcp [8/256/0/180] [ 5679.544402] Key type lgssc registered [ 5681.096376] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5691.060125] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5691.105201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5692.704731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5698.178623] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 5702.119075] LustreError: 191143:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5702.142649] LustreError: 191143:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5707.245905] LustreError: 191144:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5708.278087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5708.563491] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5713.548627] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 5721.877894] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5728.595787] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5736.495719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5736.722161] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5736.727491] LustreError: 193007:0:(llog.c:1687:llog_backup()) MGC192.168.201.102@tcp: failed to backup log lustre-OST0000: rc = -28 [ 5736.736510] Lustre: 193007:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.201.102@tcp: can't backup local config lustre-OST0000: rc = -28 [ 5736.858429] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5743.445846] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 5750.517485] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5753.313195] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5753.319210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5753.325389] Lustre: Skipped 1 previous similar message [ 5753.332914] Lustre: Skipped 2 previous similar messages [ 5754.340270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5754.357489] Lustre: Skipped 2 previous similar messages [ 5758.824983] Lustre: server umount lustre-MDT0000 complete [ 5759.459212] LustreError: 193032:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5759.478224] LustreError: 193032:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5762.670783] LustreError: 191124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782486141 with bad export cookie 10670817186691326487 [ 5762.683660] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5762.688146] LustreError: 191124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5764.579161] LustreError: 191140:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5764.580778] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5764.622715] Lustre: Skipped 1 previous similar message [ 5764.629881] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5764.639585] LustreError: 191873:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000f7adc809 x1869072098685568/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5764.794991] Lustre: server umount lustre-MDT0001 complete [ 5769.968818] Lustre: server umount lustre-OST0000 complete [ 5777.100808] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_hostid [ 5785.522441] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 5811.367240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5811.535265] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5811.552135] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5811.621901] Lustre: lustre-MDT0000: new disk, initializing [ 5811.690132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5811.701446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5814.689465] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 5822.012790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5822.077552] Lustre: 196525:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5822.118429] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5822.124220] Lustre: Skipped 1 previous similar message [ 5822.126370] LustreError: 197310:0:(llog.c:1687:llog_backup()) MGC192.168.201.102@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 5822.137188] LustreError: 197310:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 5822.141260] Lustre: 197310:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.102@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 5822.157853] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5822.160962] Lustre: Skipped 1 previous similar message [ 5822.216314] Lustre: lustre-MDT0001: new disk, initializing [ 5822.291449] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5822.311971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5822.327949] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5824.752386] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 5830.454421] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5833.815217] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5838.080992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5838.252096] Lustre: *** cfs_fail_loc=131e, val=0*** [ 5838.254631] Lustre: Skipped 1 previous similar message [ 5838.256921] LustreError: 198470:0:(llog.c:1687:llog_backup()) MGC192.168.201.102@tcp: failed to backup log lustre-OST0000: rc = -28 [ 5838.262455] LustreError: 198470:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 5838.267787] Lustre: 198470:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.102@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 5838.273181] Lustre: 198470:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 5838.288353] Lustre: lustre-OST0000: new disk, initializing [ 5838.290709] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5838.295442] Lustre: 198493:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5838.364643] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5842.261534] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 5844.504498] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5844.509064] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5844.582472] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5848.259336] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5850.870419] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5850.986327] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5853.641084] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5853.758224] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5859.813077] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5859.816699] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5859.821812] LustreError: Skipped 1 previous similar message [ 5859.823312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5859.847218] Lustre: Skipped 1 previous similar message [ 5860.979640] Lustre: server umount lustre-OST0000 complete [ 5864.931432] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5864.940981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5864.944522] Lustre: Skipped 2 previous similar messages [ 5869.856546] Lustre: server umount lustre-MDT0000 complete [ 5870.049117] LustreError: 196535:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5870.060202] LustreError: 196535:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5872.333720] LustreError: 197323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782486251 with bad export cookie 10670817186691327208 [ 5872.338218] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5872.344589] LustreError: 197323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5872.523932] Lustre: server umount lustre-MDT0001 complete [ 5877.783956] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 11:04:16 (1782486256) [ 5888.073821] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 5894.189660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5894.414355] LustreError: 201614:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5894.446803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5896.969130] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 5902.048580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5904.584133] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 5906.371326] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5910.379837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5914.273689] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 5919.840917] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5919.975479] Lustre: lustre-OST0001: new disk, initializing [ 5919.977695] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5919.986724] Lustre: 203922:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5923.560385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5923.566635] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5923.615625] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5923.778121] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 5930.837054] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5935.406970] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5936.403816] Lustre: DEBUG MARKER: ADD OST3 [ 5938.921517] loop: module loaded [ 5938.949820] loop0: detected capacity change from 0 to 204800000 [ 5939.114404] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5939.123102] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5939.133634] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5942.510255] loop0: detected capacity change from 0 to 204800000 [ 5944.681039] Lustre: DEBUG MARKER: STOP OST3 [ 5945.604975] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5945.719854] Lustre: lustre-OST0003: new disk, initializing [ 5945.727592] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 5945.731870] Lustre: 205787:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5945.787498] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 5945.790611] Lustre: Skipped 3 previous similar messages [ 5947.702984] Lustre: *** cfs_fail_loc=2109, val=2*** [ 5947.749608] Lustre: Failing over lustre-OST0003 [ 5947.752434] LustreError: lustre-OST0003-osc-MDT0000: operation ost_get_info to node 0@lo failed: rc = -19 [ 5947.752587] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5947.752592] Lustre: Skipped 1 previous similar message [ 5947.768699] LustreError: Skipped 1 previous similar message [ 5947.823734] Lustre: server umount lustre-OST0003 complete [ 5948.869614] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 5949.727358] LustreError: 206126:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 5952.903626] LustreError: 203098:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.201.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5952.914697] LustreError: 203098:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5954.111584] Lustre: DEBUG MARKER: START OST3 again [ 5957.736229] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5957.901226] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 5957.994187] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 5959.412695] Lustre: lustre-OST0003: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 5959.413042] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5959.435643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 5959.440385] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 5959.468444] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000401 [ 5961.321532] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 5976.767159] Lustre: Failing over lustre-OST0003 [ 5976.826321] Lustre: server umount lustre-OST0003 complete [ 5978.606186] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5978.621075] Lustre: Skipped 2 previous similar messages [ 5988.840973] LustreError: 203097:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5988.848977] LustreError: 203097:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 5993.958712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5993.963273] Lustre: Skipped 1 previous similar message [ 5996.935702] Lustre: server umount lustre-MDT0000 complete [ 5999.323988] LustreError: 201597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782486378 with bad export cookie 10670817186691328083 [ 5999.325145] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5999.333900] LustreError: 201597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6004.195224] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6004.204150] Lustre: Skipped 5 previous similar messages [ 6005.701522] Lustre: server umount lustre-MDT0001 complete [ 6008.146456] Lustre: server umount lustre-OST0000 complete [ 6010.560585] Lustre: server umount lustre-OST0001 complete [ 6014.223605] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_hostid [ 6018.791179] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 6040.334430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6040.463617] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6040.478570] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6040.526301] Lustre: lustre-MDT0000: new disk, initializing [ 6040.572677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6040.575327] Lustre: Skipped 1 previous similar message [ 6040.581524] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6042.461438] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 6048.193769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6048.253507] Lustre: 211219:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6048.377649] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6050.476312] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 6056.232826] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6058.968103] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6062.586804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6062.773455] Lustre: lustre-OST0000: new disk, initializing [ 6062.775806] Lustre: Skipped 1 previous similar message [ 6062.778245] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6062.781603] Lustre: Skipped 2 previous similar messages [ 6062.785933] Lustre: 213186:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6063.829913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6063.835160] Lustre: Skipped 1 previous similar message [ 6063.839550] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6063.905928] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6065.956717] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 6071.412750] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6073.673715] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6073.806776] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6076.048763] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6076.170208] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6079.458058] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6079.459239] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6079.460890] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6079.460896] Lustre: Skipped 3 previous similar messages [ 6079.461769] LustreError: Skipped 1 previous similar message [ 6079.476374] Lustre: Skipped 1 previous similar message [ 6083.456896] Lustre: server umount lustre-OST0000 complete [ 6089.184995] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6091.611154] Lustre: server umount lustre-MDT0000 complete [ 6093.472327] LustreError: 211212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782486472 with bad export cookie 10670817186691330386 [ 6093.476529] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6093.476891] LustreError: 211212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6093.645457] Lustre: server umount lustre-MDT0001 complete [ 6095.538714] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 11:07:54 (1782486474) [ 6103.564383] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_hostid [ 6108.862176] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 6126.906464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6127.040157] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6127.084061] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6127.124864] Lustre: lustre-MDT0000: new disk, initializing [ 6127.178125] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6129.122980] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 6134.507326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6134.569153] Lustre: 217787:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6134.715605] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6137.165163] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 6142.243425] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6144.888489] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6148.393113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6148.569525] Lustre: 219752:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6150.246770] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6152.157062] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 6157.664400] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6159.976362] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6160.089391] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6162.169767] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6162.297608] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6165.478888] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6165.479985] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6165.481370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6165.481376] Lustre: Skipped 3 previous similar messages [ 6165.486052] Lustre: Skipped 3 previous similar messages [ 6165.490560] LustreError: Skipped 1 previous similar message [ 6169.482604] Lustre: server umount lustre-OST0000 complete [ 6175.713081] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6177.772561] Lustre: server umount lustre-MDT0000 complete [ 6179.562569] LustreError: 217778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782486558 with bad export cookie 10670817186691331268 [ 6179.564983] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6179.568931] LustreError: 217778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6179.735135] Lustre: server umount lustre-MDT0001 complete [ 6183.973234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6184.234435] LustreError: 221964:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6184.253443] LustreError: 221964:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 6184.284716] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6184.287598] Lustre: Skipped 5 previous similar messages [ 6186.409667] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 6191.281095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6193.643371] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 6197.437710] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6200.142326] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6203.424832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6206.516146] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 6210.073475] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6225.378144] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6229.350468] Lustre: server umount lustre-OST0000 complete [ 6232.545193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6232.549857] Lustre: Skipped 7 previous similar messages [ 6245.343146] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6245.452261] Lustre: server umount lustre-MDT0000 complete [ 6247.477864] LustreError: 224561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782486626 with bad export cookie 10670817186691332150 [ 6247.480386] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6247.485294] LustreError: 224561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6247.635769] Lustre: server umount lustre-MDT0001 complete [ 6352.313702] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 6357.429871] Key type lgssc unregistered [ 6357.963510] LNet: 225649:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6357.974796] LNetError: 225649:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6358.004383] LNet: Removed LNI 192.168.201.102@tcp [ 6359.680162] Key type .llcrypt unregistered [ 6359.681863] Key type ._llcrypt unregistered [ 6390.664526] Key type ._llcrypt registered [ 6390.666596] Key type .llcrypt registered [ 6390.752330] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_hostid [ 6407.627331] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 6408.581822] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6408.777305] alg: No test for adler32 (adler32-zlib) [ 6409.835276] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 6410.090667] LNet: Added LNI 192.168.201.102@tcp [8/256/0/180] [ 6411.791201] Key type lgssc registered [ 6412.743786] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6445.853949] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6445.876413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6447.087870] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6447.123549] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6447.221340] Lustre: lustre-MDT0000: new disk, initializing [ 6447.321990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6447.341469] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6451.762835] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 6461.368681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6461.472751] Lustre: 229007:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6461.542609] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6461.545941] Lustre: Skipped 1 previous similar message [ 6461.618241] Lustre: lustre-MDT0001: new disk, initializing [ 6461.733551] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6461.792210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6461.809647] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6466.121821] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 6474.940334] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6480.950653] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6487.854203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6488.104212] Lustre: lustre-OST0000: new disk, initializing [ 6488.107700] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6488.115228] Lustre: 230975:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6488.257592] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6495.645883] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 6497.372198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6497.384666] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6497.489933] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6505.453184] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6510.330620] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6510.479197] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6514.823000] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6515.023406] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6518.240851] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6518.252346] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6518.257766] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6518.282379] Lustre: Skipped 1 previous similar message [ 6523.099769] Lustre: server umount lustre-OST0000 complete [ 6528.484454] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6528.485783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6528.486288] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6528.486292] LustreError: Skipped 1 previous similar message [ 6528.498105] Lustre: Skipped 1 previous similar message [ 6528.505478] Lustre: Skipped 1 previous similar message [ 6532.558815] Lustre: server umount lustre-MDT0000 complete [ 6535.692739] LustreError: 228998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782486914 with bad export cookie 4486097781213699212 [ 6535.694215] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6535.701683] LustreError: 228998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6536.013482] Lustre: server umount lustre-MDT0001 complete [ 6553.839507] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 6563.162627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6563.517797] LustreError: 233908:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6563.532623] LustreError: 233908:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6563.568772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6567.626475] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 6568.930095] LustreError: 233909:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6574.052249] LustreError: 233908:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6575.389562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6575.647762] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6579.020506] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 6581.383410] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6586.862030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6587.247555] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6592.320959] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 6599.671623] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6599.849808] Lustre: lustre-OST0001: new disk, initializing [ 6599.855659] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6599.860413] Lustre: 236214:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6605.979487] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 6607.956047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6607.961537] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6608.060284] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6615.964480] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6620.804596] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6630.248043] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 11:16:48 (1782487008) [ 6631.905334] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6631.910364] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6631.920146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6631.923620] Lustre: Skipped 1 previous similar message [ 6633.953149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6633.957226] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6633.969291] Lustre: Skipped 2 previous similar messages [ 6633.988152] Lustre: Skipped 2 previous similar messages [ 6637.926593] Lustre: server umount lustre-MDT0000 complete [ 6639.074793] LustreError: 233908:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6639.105270] LustreError: 233908:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6641.105534] LustreError: 234642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782487020 with bad export cookie 4486097781213700094 [ 6641.113203] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6641.116282] LustreError: 234642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6641.536478] Lustre: server umount lustre-MDT0001 complete [ 6655.518346] Lustre: server umount lustre-OST0000 complete [ 6669.062953] Lustre: server umount lustre-OST0001 complete [ 6674.900793] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_hostid [ 6683.967609] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 6713.152198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6713.291827] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6713.320804] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6713.400754] Lustre: lustre-MDT0000: new disk, initializing [ 6713.476715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6713.481732] Lustre: Skipped 1 previous similar message [ 6713.500149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6717.703330] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 6726.516568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6726.619769] Lustre: 240158:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6726.664306] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6726.666772] Lustre: Skipped 1 previous similar message [ 6726.720409] Lustre: lustre-MDT0001: new disk, initializing [ 6726.819916] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6726.826685] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6730.418292] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 6738.036312] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6742.276673] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6746.664776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6746.866106] Lustre: lustre-OST0000: new disk, initializing [ 6746.869745] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6746.877579] Lustre: 242124:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6746.946267] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6746.951539] Lustre: Skipped 1 previous similar message [ 6747.954421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6747.967717] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6748.031129] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6751.315764] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 6757.687428] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6760.169380] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6760.312780] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6762.895577] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6763.044877] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6768.616510] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6768.616639] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6768.630867] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6768.641582] Lustre: Skipped 1 previous similar message [ 6770.433549] Lustre: server umount lustre-OST0000 complete [ 6773.215873] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6773.219208] LustreError: Skipped 1 previous similar message [ 6778.604318] Lustre: server umount lustre-MDT0000 complete [ 6778.848537] LustreError: 240169:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6778.855442] LustreError: 240169:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6780.322861] LustreError: 242123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782487159 with bad export cookie 4486097781213702229 [ 6780.327422] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6780.330236] LustreError: 242123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6780.496345] Lustre: server umount lustre-MDT0001 complete [ 6784.500151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6786.982908] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 6790.116328] LustreError: 244339:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6790.124981] LustreError: 244339:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6791.243241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6793.652600] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 6797.350167] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6800.216275] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6803.747291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6806.732134] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 6809.982790] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6953.440728] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6953.445160] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6953.449416] Lustre: Skipped 3 previous similar messages [ 6953.452225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6953.454502] Lustre: Skipped 2 previous similar messages [ 6959.451033] Lustre: server umount lustre-OST0000 complete [ 6967.013711] Lustre: server umount lustre-MDT0000 complete [ 6968.524948] LustreError: 244319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782487347 with bad export cookie 4486097781213703111 [ 6968.527047] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6968.530308] LustreError: 244319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6968.653886] Lustre: server umount lustre-MDT0001 complete [ 7075.173592] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 7076.231309] Key type lgssc unregistered [ 7076.362434] LNet: 247941:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7076.365162] LNetError: 247941:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7076.371334] LNet: Removed LNI 192.168.201.102@tcp [ 7076.707126] Key type .llcrypt unregistered [ 7076.708555] Key type ._llcrypt unregistered [ 7085.959000] Key type ._llcrypt registered [ 7085.960248] Key type .llcrypt registered [ 7085.999510] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_hostid [ 7092.691287] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 7093.118223] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7093.136373] alg: No test for adler32 (adler32-zlib) [ 7094.014712] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 7094.111755] LNet: Added LNI 192.168.201.102@tcp [8/256/0/180] [ 7095.695219] Key type lgssc registered [ 7096.087193] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7113.213103] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 7116.672853] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7116.680029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7117.759800] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7117.769983] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7117.804036] Lustre: lustre-MDT0000: new disk, initializing [ 7117.822868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7117.828388] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7119.174497] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 7123.364557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7123.407398] Lustre: 251984:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7123.443926] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7123.446113] Lustre: Skipped 1 previous similar message [ 7123.480626] Lustre: lustre-MDT0001: new disk, initializing [ 7123.512079] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7123.527722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7123.531064] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7124.891451] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 7127.169951] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7129.458576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7129.584259] Lustre: lustre-OST0000: new disk, initializing [ 7129.586353] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7129.590385] Lustre: 253614:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7129.626290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7131.624678] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 7131.968705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7131.973863] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7132.027433] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7135.973494] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7136.053292] Lustre: lustre-OST0001: new disk, initializing [ 7136.055519] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7136.059296] Lustre: 254486:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7136.097118] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7138.097872] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 7140.867539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7140.872537] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7140.900902] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7143.775038] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7147.681768] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7151.075524] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7151.078197] LustreError: 255014:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7151.081818] Lustre: Skipped 3 previous similar messages [ 7151.088385] LustreError: 255014:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7151.092343] Lustre: server umount lustre-MDT0000 complete [ 7152.430440] LustreError: 251975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782487531 with bad export cookie 6495191081692934991 [ 7152.433876] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7152.435028] LustreError: 251975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7156.192437] LustreError: 251990:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7156.192924] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7156.194437] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7156.198894] LustreError: 251990:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7156.203930] Lustre: Skipped 1 previous similar message [ 7157.216060] LustreError: 251989:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7157.216417] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7157.222859] LustreError: 251989:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7157.225256] Lustre: Skipped 2 previous similar messages [ 7158.660113] Lustre: server umount lustre-MDT0001 complete [ 7160.122207] Lustre: server umount lustre-OST0000 complete [ 7161.458392] Lustre: server umount lustre-OST0001 complete [ 7166.311814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7169.766508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7172.625241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7175.688130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7181.891076] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 7185.388166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7185.394433] Lustre: lustre-MDT0000: reset Object Index mappings [ 7185.518833] LustreError: 259484:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7185.532615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7186.932316] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 7189.841750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7189.850286] Lustre: lustre-MDT0001: reset Object Index mappings [ 7191.366051] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 7192.294168] Lustre: 260627:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7194.641722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7194.770331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7194.773309] Lustre: Skipped 1 previous similar message [ 7195.811286] LustreError: 260979:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7195.814139] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7196.717525] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 7199.754550] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7200.875119] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 7201.849165] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 7204.999059] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7206.160279] Lustre: 262494:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7210.463960] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7210.467205] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7210.471910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7213.465704] Lustre: server umount lustre-MDT0000 complete [ 7214.764479] LustreError: 259466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782487593 with bad export cookie 6495191081692941753 [ 7214.768269] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7214.769991] LustreError: 259466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7214.888340] Lustre: server umount lustre-MDT0001 complete [ 7226.818237] Lustre: server umount lustre-OST0000 complete [ 7237.277101] Lustre: server umount lustre-OST0001 complete [ 7243.870480] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 11:27:02 (1782487622) [ 7248.980416] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_hostid [ 7252.090523] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 7263.888956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7263.971288] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7263.982903] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7264.020222] Lustre: lustre-MDT0000: new disk, initializing [ 7264.047780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7264.050167] Lustre: Skipped 1 previous similar message [ 7264.055663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7265.426519] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 7269.664019] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7269.704131] Lustre: 266737:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7269.808408] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7271.165590] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 7274.851404] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7276.708635] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7278.993219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7279.088590] Lustre: lustre-OST0000: new disk, initializing [ 7279.090321] Lustre: Skipped 1 previous similar message [ 7279.092269] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7279.094601] Lustre: Skipped 2 previous similar messages [ 7279.096566] Lustre: 268704:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7280.571044] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7280.574143] Lustre: Skipped 1 previous similar message [ 7280.576280] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7280.614895] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7281.058798] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 7284.858928] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7286.285818] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7286.358929] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7287.764874] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7287.830527] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7290.849345] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7290.849520] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7290.851721] LustreError: Skipped 1 previous similar message [ 7290.852305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7290.855641] Lustre: Skipped 4 previous similar messages [ 7290.856477] Lustre: Skipped 4 previous similar messages [ 7294.547857] Lustre: server umount lustre-OST0000 complete [ 7295.968162] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7295.972898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7295.975270] Lustre: Skipped 1 previous similar message [ 7300.576023] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7306.210761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7306.212793] Lustre: Skipped 4 previous similar messages [ 7310.304467] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7310.377581] Lustre: server umount lustre-MDT0000 complete [ 7311.775172] LustreError: 266729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782487690 with bad export cookie 6495191081692943825 [ 7311.777823] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7311.779385] LustreError: 266729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7311.886659] Lustre: server umount lustre-MDT0001 complete [ 7318.088181] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 7321.572448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7321.700928] LustreError: 271634:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7321.714338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7321.717140] Lustre: Skipped 2 previous similar messages [ 7323.098616] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 7326.151972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7327.704727] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 7328.742137] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7331.140254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7333.261499] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 7336.232410] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7336.296791] Lustre: lustre-OST0001: new disk, initializing [ 7336.298747] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7336.302027] Lustre: 273941:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7337.352337] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7337.356651] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7337.391436] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7338.255185] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 7342.740064] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7343.987361] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7345.877122] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 7345.882315] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 7351.775997] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7351.780295] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7351.785642] Lustre: Skipped 1 previous similar message [ 7351.788120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7351.790182] Lustre: Skipped 2 previous similar messages [ 7353.881484] Lustre: server umount lustre-MDT0000 complete [ 7355.151359] LustreError: 274704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782487734 with bad export cookie 6495191081692944707 [ 7355.154929] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7355.156427] LustreError: 274704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7355.272525] Lustre: server umount lustre-MDT0001 complete [ 7367.115404] Lustre: server umount lustre-OST0000 complete [ 7379.627789] Lustre: server umount lustre-OST0001 complete [ 7407.608510] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 7436.925202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7437.933432] LustreError: 276999:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7438.042018] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7438.059681] Lustre: Skipped 3 previous similar messages [ 7444.539740] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 7461.274604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7469.607776] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 7473.946816] Lustre: 278142:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7483.216960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7486.824125] LustreError: 278499:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7486.841781] LustreError: 278499:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 7493.163350] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 7507.547446] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 7509.577628] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7515.171724] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 7519.030552] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 7528.268287] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7532.010899] Lustre: 280023:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 7538.037647] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 7538.065226] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 7548.529861] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 11:32:06 (1782487926) [ 7564.256376] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7564.274081] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7564.288591] Lustre: Skipped 3 previous similar messages [ 7564.292450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7564.295280] Lustre: Skipped 4 previous similar messages [ 7569.436737] Lustre: server umount lustre-MDT0000 complete [ 7571.439757] LustreError: 278995:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7571.476901] LustreError: 278995:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 7573.890511] LustreError: 276980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782487952 with bad export cookie 6495191081692951014 [ 7573.892187] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7573.907603] LustreError: 276980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7588.320292] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7588.586422] Lustre: server umount lustre-MDT0001 complete [ 7600.061389] Lustre: server umount lustre-OST0000 complete [ 7604.627916] Lustre: server umount lustre-OST0001 complete [ 7612.439831] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_hostid [ 7622.288493] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 7671.817856] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 7684.059912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7684.411585] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7684.454939] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7684.536105] Lustre: lustre-MDT0000: new disk, initializing [ 7684.620801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7684.628973] Lustre: Skipped 3 previous similar messages [ 7684.641484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7689.482528] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 7703.260077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7703.542611] Lustre: 284265:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7704.148870] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7709.961626] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 7723.983253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7724.311592] Lustre: 285900:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7725.863832] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7730.602482] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 7742.086239] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7742.277381] Lustre: 286772:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7743.584460] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7743.587558] Lustre: Skipped 1 previous similar message [ 7743.720733] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7749.729317] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 7760.254671] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7763.842727] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7763.848189] Lustre: Skipped 1 previous similar message [ 7780.778796] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 11:35:58 (1782488158) [ 7784.935377] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7784.946487] Lustre: Skipped 5 previous similar messages [ 7784.954674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7784.958813] Lustre: Skipped 13 previous similar messages [ 7788.971978] Lustre: server umount lustre-MDT0000 complete [ 7790.052467] LustreError: 285133:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7790.077878] LustreError: 285133:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 7793.264584] LustreError: 285060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782488172 with bad export cookie 6495191081692958882 [ 7793.265779] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7793.287772] LustreError: 285060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7793.602962] Lustre: server umount lustre-MDT0001 complete [ 7808.958195] Lustre: server umount lustre-OST0000 complete [ 7811.874137] Lustre: 249498:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782488174/real 1782488174] req@ffff8adddfb90700 x1869073584823808/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782488190 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7813.855537] Lustre: server umount lustre-OST0001 complete [ 7823.773682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7829.357396] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 7838.566142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7843.579767] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 7852.174286] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7858.307933] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7865.791188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7874.094420] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 7884.294555] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7906.809133] loop1: detected capacity change from 0 to 204800000 [ 7907.297921] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7907.327914] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7907.356968] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7913.937927] loop1: detected capacity change from 0 to 204800000 [ 7921.468548] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7921.608711] LustreError: 289009:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) fs20518: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 7921.653451] LustreError: 289009:0:(mgs_llog.c:731:mgs_check_index()) Can't get db for fs20518 [ 7921.660375] LustreError: 289009:0:(mgs_handler.c:517:mgs_target_reg()) fs20518-MDT0001: Error checking target index: rc = -13 [ 7921.674853] LustreError: fs20518-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 7921.682634] LustreError: 292742:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 7921.692702] LustreError: 292742:0:(tgt_mount.c:2129:server_put_super()) no obd fs20518-MDT0001 [ 7921.698077] LustreError: 292742:0:(tgt_mount.c:116:server_deregister_mount()) fs20518-MDT0001 not registered [ 7921.754687] Lustre: server umount fs20518-MDT0001 complete [ 7921.758067] LustreError: 292742:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 7933.347254] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7933.669196] Lustre: Setting parameter fs20518-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs20518-MDT0001 [ 7933.745838] Lustre: srv-fs20518-MDT0001: No data found on store. Initialize space. [ 7933.762496] Lustre: Skipped 4 previous similar messages [ 7933.851081] Lustre: fs20518-MDT0001: new disk, initializing [ 7933.854169] Lustre: Skipped 3 previous similar messages [ 7933.955229] Lustre: srv-fs20518-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 7933.960097] Lustre: 293209:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs20518-MDT0001-mdtlov: can't create new llog: rc = -115 [ 7933.966205] LustreError: 293209:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs20518-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 7939.119244] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 7946.757589] Lustre: server umount fs20518-MDT0001 complete [ 7950.307409] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [ 7964.507985] loop1: detected capacity change from 0 to 204800000 [ 7972.385810] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7972.974617] Lustre: fs20518-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7972.979952] Lustre: Skipped 7 previous similar messages [ 7972.992227] Lustre: 294777:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs20518-MDT0001-mdtlov: can't create new llog: rc = -115 [ 7973.005331] LustreError: 294777:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs20518-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 7978.359768] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 7984.184877] Lustre: server umount fs20518-MDT0001 complete [ 7996.387486] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7996.398518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7996.423804] Lustre: Skipped 4 previous similar messages [ 7996.440881] Lustre: Skipped 2 previous similar messages [ 8002.294111] Lustre: server umount lustre-MDT0000 complete [ 8015.843057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8016.400873] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8022.028577] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8022.779323] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 8032.572798] loop1: detected capacity change from 0 to 204800000 [ 8040.291543] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8041.442550] Lustre: 297225:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs20518-MDT0001-mdtlov: can't create new llog: rc = -115 [ 8041.458066] LustreError: 297225:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs20518-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 8047.615723] LustreError: 290849:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8047.648606] LustreError: 290849:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 108 previous similar messages [ 8047.682467] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 8053.127735] Lustre: server umount fs20518-MDT0001 complete [ 8083.435220] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8083.441525] LustreError: Skipped 1 previous similar message [ 8086.363923] Lustre: server umount lustre-OST0000 complete [ 8095.952962] Lustre: server umount lustre-MDT0000 complete [ 8099.084572] LustreError: 295769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782488478 with bad export cookie 6495191081692961563 [ 8099.089753] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8099.092455] LustreError: 295769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8099.826013] Lustre: server umount lustre-MDT0001 complete [ 8107.872235] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 8109.730231] Key type lgssc unregistered [ 8109.959803] LNet: 299573:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8109.968144] LNetError: 299573:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8109.989165] LNet: Removed LNI 192.168.201.102@tcp [ 8110.595181] Key type .llcrypt unregistered [ 8110.597152] Key type ._llcrypt unregistered [ 8130.068496] Key type ._llcrypt registered [ 8130.072393] Key type .llcrypt registered [ 8130.196272] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 8131.294551] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8131.511735] alg: No test for adler32 (adler32-zlib) [ 8132.678428] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 8132.875101] LNet: Added LNI 192.168.201.102@tcp [8/256/0/180] [ 8134.535151] Key type lgssc registered [ 8135.346908] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8139.162512] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8139.178449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8140.394286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8142.229839] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 8146.091189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8146.321957] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8148.074528] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 8151.221732] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8153.765928] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8156.851297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8157.018942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8159.713856] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 8163.174968] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8165.217568] LustreError: 303341:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8165.225713] LustreError: 303341:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8169.821717] LustreError: 303341:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8171.490347] LustreError: 302669:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8171.499240] LustreError: 302669:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8174.954554] LustreError: 303341:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8180.077795] LustreError: 302671:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8180.091279] LustreError: 302671:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8180.889920] loop1: detected capacity change from 0 to 204800000 [ 8180.955551] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8180.961166] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8180.966797] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8183.080021] loop1: detected capacity change from 0 to 204800000 [ 8185.340243] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8185.403807] Lustre: fs11331-OST0000: new disk, initializing [ 8185.405741] Lustre: srv-fs11331-OST0000: No data found on store. Initialize space. [ 8185.408888] Lustre: 304598:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8185.436853] Lustre: fs11331-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8187.722371] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 8190.316044] LustreError: 302669:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8190.321860] LustreError: 302669:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 8191.852291] Lustre: server umount fs11331-OST0000 complete [ 8196.695481] loop1: detected capacity change from 0 to 204800000 [ 8199.164598] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8199.244959] Lustre: fs11331-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8201.384210] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 8204.238847] loop2: detected capacity change from 0 to 204800000 [ 8204.318209] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8204.322301] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8204.326243] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8205.932408] loop2: detected capacity change from 0 to 204800000 [ 8207.329864] LustreError: 303341:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8207.334983] LustreError: 303341:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 8207.679513] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8207.711215] LustreError: fs11331-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 8207.715458] LustreError: fs11331-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 8207.718211] LustreError: 307633:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 8207.720475] LustreError: 307633:0:(tgt_mount.c:2129:server_put_super()) no obd fs11331-OST0001 [ 8207.723432] LustreError: 307633:0:(tgt_mount.c:116:server_deregister_mount()) fs11331-OST0001 not registered [ 8207.735442] Lustre: server umount fs11331-OST0001 complete [ 8207.736683] LustreError: 307633:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 8211.084501] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8211.126856] Lustre: fs11331-OST0001: new disk, initializing [ 8211.129376] Lustre: srv-fs11331-OST0001: No data found on store. Initialize space. [ 8211.132805] Lustre: 308181:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8211.154823] Lustre: fs11331-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8212.991343] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 8216.349664] Lustre: server umount fs11331-OST0000 complete [ 8219.383743] Lustre: server umount fs11331-OST0001 complete [ 8223.291890] loop1: detected capacity change from 0 to 204800000 [ 8225.019711] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8226.847189] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 8228.989529] loop2: detected capacity change from 0 to 204800000 [ 8230.743408] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8230.806734] Lustre: fs11331-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8230.809668] Lustre: Skipped 1 previous similar message [ 8232.627138] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 8235.426742] Lustre: server umount fs11331-OST0000 complete [ 8238.366139] Lustre: server umount fs11331-OST0001 complete [ 8242.144411] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8242.147043] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8242.151307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8243.168915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8243.169425] LustreError: 303341:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8243.169450] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8243.169478] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8243.180770] LustreError: 303341:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 19 previous similar messages [ 8247.952636] Lustre: server umount lustre-OST0000 complete [ 8253.408948] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8253.409563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8253.412572] Lustre: Skipped 1 previous similar message [ 8253.417077] Lustre: Skipped 2 previous similar messages [ 8255.321338] Lustre: server umount lustre-MDT0000 complete [ 8256.611698] LustreError: 300823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782488635 with bad export cookie 6071796803386283819 [ 8256.614049] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8256.617128] LustreError: 300823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8256.744696] Lustre: server umount lustre-MDT0001 complete [ 8260.821887] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 8261.913673] Key type lgssc unregistered [ 8262.055398] LNet: 314090:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8262.059789] LNetError: 314090:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8262.067343] LNet: Removed LNI 192.168.201.102@tcp [ 8262.386136] Key type .llcrypt unregistered [ 8262.387962] Key type ._llcrypt unregistered [ 8273.311053] Key type ._llcrypt registered [ 8273.312534] Key type .llcrypt registered [ 8273.346958] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_hostid [ 8279.243612] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 8279.521289] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8279.527082] alg: No test for adler32 (adler32-zlib) [ 8280.374024] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 8280.458467] LNet: Added LNI 192.168.201.102@tcp [8/256/0/180] [ 8282.039173] Key type lgssc registered [ 8282.371178] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8293.219672] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8293.226902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8294.326666] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8294.338563] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8294.373146] Lustre: lustre-MDT0000: new disk, initializing [ 8294.393972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8294.399641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8295.777870] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 8300.032386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8300.082821] Lustre: 317919:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8300.114839] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8300.117856] Lustre: Skipped 1 previous similar message [ 8300.167702] Lustre: lustre-MDT0001: new disk, initializing [ 8300.212098] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8300.226663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8300.230783] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8301.749834] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 8305.533266] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8307.266514] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8309.496293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8309.622554] Lustre: lustre-OST0000: new disk, initializing [ 8309.624293] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8309.625528] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 8309.631348] Lustre: 319890:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8309.661660] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8311.545758] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 8314.863109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8314.866277] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8314.874130] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 8315.101962] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8315.737916] Lustre: Mounted lustre-client [ 8316.041465] Lustre: Unmounted lustre-client [ 8319.967793] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8319.971354] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8319.978379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8323.042835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8323.043626] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8323.047813] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8324.822736] Lustre: server umount lustre-OST0000 complete [ 8326.111897] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8326.114463] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8326.118809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8326.120734] Lustre: Skipped 1 previous similar message [ 8328.160927] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8332.135867] Lustre: server umount lustre-MDT0000 complete [ 8333.417249] LustreError: 317913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782488712 with bad export cookie 9499269130517551285 [ 8333.420626] LustreError: 317913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8333.421035] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8333.537446] Lustre: server umount lustre-MDT0001 complete [ 8337.381469] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 8338.439851] Key type lgssc unregistered [ 8373.407625] LNet: 322043:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8373.413008] LNetError: 322043:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8447.141609] LNet: Removed LNI 192.168.201.102@tcp [ 8447.953176] Key type .llcrypt unregistered [ 8447.954996] Key type ._llcrypt unregistered [ 8459.277526] Key type ._llcrypt registered [ 8459.279381] Key type .llcrypt registered [ 8459.361114] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_hostid [ 8467.389534] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 8468.013142] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8468.040334] alg: No test for adler32 (adler32-zlib) [ 8468.951143] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 8469.099720] LNet: Added LNI 192.168.201.102@tcp [8/256/0/180] [ 8470.727234] Key type lgssc registered [ 8471.519419] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8489.673879] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 11:47:48 (1782488868) [ 8493.803101] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8493.821641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8494.942875] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8494.960375] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8495.010612] Lustre: lustre-MDT0000: new disk, initializing [ 8495.058449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8495.100122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8498.029870] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 8505.236651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8505.320283] Lustre: 325248:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8505.368890] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8505.371693] Lustre: Skipped 1 previous similar message [ 8505.430892] Lustre: lustre-MDT0001: new disk, initializing [ 8505.502605] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8505.521430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8505.533190] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8508.154459] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 8514.174665] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8517.667293] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8521.833393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8522.002152] Lustre: lustre-OST0000: new disk, initializing [ 8522.005817] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8522.010220] Lustre: 327215:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8522.067990] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8525.340589] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 8530.970805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8530.979312] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8531.069142] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8531.159363] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8541.152087] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8541.153753] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8541.158217] LustreError: Skipped 1 previous similar message [ 8541.159199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8541.172745] Lustre: Skipped 1 previous similar message [ 8541.187902] Lustre: Skipped 1 previous similar message [ 8545.158205] Lustre: server umount lustre-OST0000 complete [ 8551.393488] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8551.405356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8551.410599] Lustre: Skipped 1 previous similar message [ 8553.776278] Lustre: server umount lustre-MDT0000 complete [ 8556.579420] LustreError: 325240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782488935 with bad export cookie 4778155025218379913 [ 8556.584786] LustreError: 325240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8556.585613] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8556.808242] Lustre: server umount lustre-MDT0001 complete [ 8563.817679] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 8565.382902] Key type lgssc unregistered [ 8565.582286] LNet: 329119:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8565.587149] LNetError: 329119:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8565.603691] LNet: Removed LNI 192.168.201.102@tcp [ 8566.091681] Key type .llcrypt unregistered [ 8566.093344] Key type ._llcrypt unregistered [ 8579.493970] Key type ._llcrypt registered [ 8579.496351] Key type .llcrypt registered [ 8579.581654] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 8580.338528] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8580.419569] alg: No test for adler32 (adler32-zlib) [ 8581.358388] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 8581.529705] LNet: Added LNI 192.168.201.102@tcp [8/256/0/180] [ 8583.151168] Key type lgssc registered [ 8583.898423] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8589.447651] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8589.460143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8590.811328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8593.715875] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 8599.700852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8599.929101] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8602.180183] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 8606.507763] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8609.668915] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8613.564026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8613.819682] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8617.640872] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 8622.703931] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8634.289716] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8635.523536] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8640.897050] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.2@tcp (stopping) [ 8640.999869] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8641.005229] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8641.509257] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8641.510699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8641.520226] Lustre: Skipped 1 previous similar message [ 8641.528222] Lustre: Skipped 3 previous similar messages [ 8644.483227] Lustre: server umount lustre-MDT0000 complete [ 8645.998717] LustreError: 330366:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8646.028440] LustreError: 330366:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8646.625457] LustreError: 330371:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8646.637591] LustreError: 330371:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8647.438723] LustreError: 330350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782489026 with bad export cookie 8251503538866134671 [ 8647.443120] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8647.450376] LustreError: 330350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8651.122233] LustreError: 331857:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8651.146630] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.2@tcp (stopping) [ 8651.745781] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8653.788082] Lustre: server umount lustre-MDT0001 complete [ 8656.388251] Lustre: server umount lustre-OST0000 complete [ 8662.198883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8665.444279] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 8667.754309] Lustre: 334548:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.102@tcp [ 8669.196245] Lustre: server umount MGS complete [ 8675.753318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8676.022743] LustreError: 335119:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8676.042875] LustreError: 335119:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8676.090299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8679.235544] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 8681.442937] LustreError: 335120:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8684.950295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8685.141550] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8687.596912] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 8692.962616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8693.161641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8695.826157] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 8700.561613] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8701.450466] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [ 8704.081258] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8708.630215] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 8711.649411] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8711.655892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8711.672860] Lustre: Skipped 1 previous similar message [ 8711.680656] Lustre: Skipped 3 previous similar messages [ 8716.769289] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8716.780339] Lustre: Skipped 1 previous similar message [ 8717.432218] Lustre: server umount lustre-OST0000 complete [ 8721.893382] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8721.907024] Lustre: Skipped 1 previous similar message [ 8727.013118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8727.031945] Lustre: Skipped 3 previous similar messages [ 8736.223134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8736.845952] Lustre: server umount lustre-MDT0000 complete [ 8737.268240] LustreError: 335120:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8737.318117] LustreError: 335120:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8744.250116] LustreError: 335100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782489123 with bad export cookie 8251503538866135511 [ 8744.259523] LustreError: 335100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8744.260981] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8745.516131] Lustre: server umount lustre-MDT0001 complete [ 8767.786295] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 8774.357668] Key type lgssc unregistered [ 8774.974992] LNet: 338580:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8774.997816] LNetError: 338580:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8775.039501] LNet: Removed LNI 192.168.201.102@tcp [ 8777.110146] Key type .llcrypt unregistered [ 8777.113608] Key type ._llcrypt unregistered [ 8799.806929] Key type ._llcrypt registered [ 8799.809313] Key type .llcrypt registered [ 8799.904886] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_hostid [ 8814.963448] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 8815.886265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8816.003219] alg: No test for adler32 (adler32-zlib) [ 8817.431343] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [ 8817.626577] LNet: Added LNI 192.168.201.102@tcp [8/256/0/180] [ 8819.279388] Key type lgssc registered [ 8820.764746] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8873.505230] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8873.564056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8875.048380] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8875.108111] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8875.233412] Lustre: lustre-MDT0000: new disk, initializing [ 8875.352799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8875.366263] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8881.325238] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 8894.351325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8894.498764] Lustre: 341747:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8894.571958] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8894.576504] Lustre: Skipped 1 previous similar message [ 8894.641609] Lustre: lustre-MDT0001: new disk, initializing [ 8894.716876] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8894.738832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8894.746678] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8899.802607] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 8910.176791] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8916.650960] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8925.128706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8925.624093] Lustre: lustre-OST0000: new disk, initializing [ 8925.629762] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8925.643873] Lustre: 343715:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8925.776374] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8932.698889] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 8934.024676] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8934.039228] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8934.269450] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8943.378187] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8948.414939] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8948.627938] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8953.422766] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8953.610098] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8959.977612] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8959.987244] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8960.003902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8960.024346] Lustre: Skipped 1 previous similar message [ 8965.092816] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8965.108714] Lustre: Skipped 1 previous similar message [ 8970.208154] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8970.216085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8970.233176] Lustre: Skipped 1 previous similar message [ 8970.599248] Lustre: server umount lustre-OST0000 complete [ 8975.330526] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8975.336245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8975.346669] Lustre: Skipped 2 previous similar messages [ 8975.372092] Lustre: Skipped 1 previous similar message [ 8980.477451] Lustre: server umount lustre-MDT0000 complete [ 8983.962986] LustreError: 341737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782489362 with bad export cookie 13077660518639921677 [ 8983.964754] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8983.979086] LustreError: 341737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8984.353861] Lustre: server umount lustre-MDT0001 complete [ 8996.843658] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 11:56:13 (1782489373) [ 9015.683570] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_hostid [ 9026.179072] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [ 9077.993658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9078.373610] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9078.411548] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9078.564560] Lustre: lustre-MDT0000: new disk, initializing [ 9078.654889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9078.682850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9083.949867] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 9095.570395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9095.730891] Lustre: 348477:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9095.817084] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9095.821552] Lustre: Skipped 1 previous similar message [ 9095.930305] Lustre: lustre-MDT0001: new disk, initializing [ 9096.016129] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9096.061148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9096.082132] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9101.718509] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 9112.994326] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9119.535291] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9126.697856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9127.018932] Lustre: lustre-OST0000: new disk, initializing [ 9127.030236] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9127.043673] Lustre: 350448:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9127.160773] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9133.966463] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [ 9135.698058] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9135.706988] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9135.802062] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9144.565206] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9182.190988] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9182.196089] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9182.200257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9182.213284] Lustre: Skipped 1 previous similar message [ 9182.259355] LustreError: Skipped 1 previous similar message [ 9185.967971] Lustre: server umount lustre-OST0000 complete [ 9192.417190] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9192.440633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9192.448718] Lustre: Skipped 2 previous similar messages [ 9196.897929] Lustre: server umount lustre-MDT0000 complete [ 9197.538430] LustreError: 348490:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9197.560895] LustreError: 348490:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 9202.150897] LustreError: 350458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782489581 with bad export cookie 13077660518639922559 [ 9202.155619] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9202.165848] LustreError: 350458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9202.765948] Lustre: server umount lustre-MDT0001 complete [ 9215.315127] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 9218.409401] Key type lgssc unregistered [ 9218.830234] LNet: 352625:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9218.844465] LNetError: 352625:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9219.889401] LNet: Removed LNI 192.168.201.102@tcp [ 9221.084978] Key type .llcrypt unregistered [ 9221.087546] Key type ._llcrypt unregistered [ 9336.921985] Key type ._llcrypt registered [ 9336.924356] Key type .llcrypt registered [ 9337.045560] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 9337.432134] Key type .llcrypt unregistered [ 9337.448375] Key type ._llcrypt unregistered [ 9436.213718] Key type ._llcrypt registered [ 9436.219344] Key type .llcrypt registered [ 9436.488650] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 9437.044494] Key type .llcrypt unregistered [ 9437.047230] Key type ._llcrypt unregistered [ 9480.533103] Key type ._llcrypt registered [ 9480.535757] Key type .llcrypt registered [ 9480.671605] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 9481.070482] Key type .llcrypt unregistered [ 9481.073511] Key type ._llcrypt unregistered [ 9527.350161] Key type ._llcrypt registered [ 9527.352947] Key type .llcrypt registered [ 9527.523879] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 9527.939970] Key type .llcrypt unregistered [ 9527.943515] Key type ._llcrypt unregistered [ 9624.432333] Key type ._llcrypt registered [ 9624.437298] Key type .llcrypt registered [ 9624.540269] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 9624.780864] Key type .llcrypt unregistered [ 9624.783399] Key type ._llcrypt unregistered [ 9660.223808] Key type ._llcrypt registered [ 9660.225552] Key type .llcrypt registered [ 9660.335987] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 9660.749429] Key type .llcrypt unregistered [ 9660.752404] Key type ._llcrypt unregistered [ 9698.159982] Key type ._llcrypt registered [ 9698.162789] Key type .llcrypt registered [ 9698.259578] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 9698.581654] Key type .llcrypt unregistered [ 9698.586236] Key type ._llcrypt unregistered [ 9742.025836] Key type ._llcrypt registered [ 9742.027255] Key type .llcrypt registered [ 9742.187718] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 9742.518880] Key type .llcrypt unregistered [ 9742.522884] Key type ._llcrypt unregistered [ 9777.959739] Key type ._llcrypt registered [ 9777.961561] Key type .llcrypt registered [ 9778.060263] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 9778.370312] Key type .llcrypt unregistered [ 9778.372274] Key type ._llcrypt unregistered [ 9811.339678] Key type ._llcrypt registered [ 9811.340985] Key type .llcrypt registered [ 9811.426477] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 9811.689344] Key type .llcrypt unregistered [ 9811.692886] Key type ._llcrypt unregistered [ 9883.306316] Key type ._llcrypt registered [ 9883.310511] Key type .llcrypt registered [ 9883.455844] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 9883.869417] Key type .llcrypt unregistered [ 9883.872604] Key type ._llcrypt unregistered [ 9923.086335] Key type ._llcrypt registered [ 9923.090311] Key type .llcrypt registered [ 9923.260775] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 9923.608470] Key type .llcrypt unregistered [ 9923.611877] Key type ._llcrypt unregistered [ 9952.381857] Key type ._llcrypt registered [ 9952.383359] Key type .llcrypt registered [ 9952.470125] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [ 9952.946409] Key type .llcrypt unregistered [ 9952.954411] Key type ._llcrypt unregistered [10026.874045] Key type ._llcrypt registered [10026.875693] Key type .llcrypt registered [10027.128085] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_hostid [10048.587829] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [10049.731056] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10049.901182] alg: No test for adler32 (adler32-zlib) [10051.100687] Lustre: Lustre: Build Version: 2.17.54_95_ga6f956b [10051.284753] LNet: Added LNI 192.168.201.102@tcp [8/256/0/180] [10052.927214] Key type lgssc registered [10054.065772] Lustre: Echo OBD driver; http://www.lustre.org/ [10092.030728] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10092.076830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10093.528877] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [10093.582337] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [10093.699517] Lustre: tst250fs-MDT0000: new disk, initializing [10093.800965] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10093.839173] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10098.676424] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [10111.889534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10112.091737] Lustre: 373009:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10112.186751] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [10112.190935] Lustre: Skipped 1 previous similar message [10112.277303] Lustre: tst250fs-MDT0001: new disk, initializing [10112.355957] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10112.381578] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10112.401520] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10118.469493] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [10123.525279] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10130.817131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10131.065886] Lustre: tst250fs-OST0000: new disk, initializing [10131.069716] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [10131.074886] Lustre: 374641:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10131.192559] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10136.634506] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10136.646473] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10136.760698] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10138.060862] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [10150.194808] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10150.416986] Lustre: tst250fs-OST0001: new disk, initializing [10150.420968] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [10150.428961] Lustre: 375513:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10150.577714] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10157.256555] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10157.278755] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10157.375449] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10159.308375] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_default_debug -1 all [10166.170737] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [10169.139717] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [10169.143988] Lustre: Skipped 2 previous similar messages [10188.258027] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10188.267915] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [10188.279848] Lustre: Skipped 1 previous similar message [10188.298912] Lustre: Skipped 3 previous similar messages [10193.248127] Lustre: server umount tst250fs-MDT0000 complete [10197.375793] LustreError: 372999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782490576 with bad export cookie 5242299311936439293 [10197.384595] LustreError: MGC192.168.201.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10197.402074] LustreError: 372999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10197.643117] Lustre: server umount tst250fs-MDT0001 complete [10211.546953] Lustre: server umount tst250fs-OST0000 complete [10214.881806] Lustre: 371232:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782490577/real 1782490577] req@ffff8adcc1b88a80 x1869076685196544/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782490593 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10214.907215] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10214.919823] Lustre: Skipped 2 previous similar messages [10216.845797] Lustre: server umount tst250fs-OST0001 complete [10235.569368] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_hostid [10245.521342] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [10290.900210] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 12:17:48 (1782490668) [10292.444920] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [10311.475478] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing set_hostid [10322.676687] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing load_modules_local [10362.925994] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 9967 sec ========== 12:19:00 (1782490740) [10364.913374] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:19:02 (1782490742) === [10368.236345] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:19:05 (1782490745) === [10412.975880] Lustre: DEBUG MARKER: oleg102-server.virtnet: executing unload_modules_local [10415.672819] Key type lgssc unregistered [10416.047990] LNet: 384645:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10416.065036] LNetError: 384645:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10416.080136] LNet: Removed LNI 192.168.201.102@tcp [10416.859161] Key type .llcrypt unregistered [10416.861836] Key type ._llcrypt unregistered