[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 565700060 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003135] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005010] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009008] pid_max: default: 32768 minimum: 301 [ 0.010119] LSM: Security Framework initializing [ 0.011040] Yama: becoming mindful. [ 0.012027] SELinux: Initializing. [ 0.013049] *** VALIDATE selinux *** [ 0.020627] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024790] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025000] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025112] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026098] *** VALIDATE tmpfs *** [ 0.027376] *** VALIDATE proc *** [ 0.028203] *** VALIDATE cgroup *** [ 0.029005] *** VALIDATE cgroup2 *** [ 0.030216] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.031112] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.032003] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.033019] Spectre V2 : User space: Vulnerable [ 0.034004] Speculative Store Bypass: Vulnerable [ 0.037133] debug: unmapping init [mem 0xffffffff8f459000-0xffffffff8f460fff] [ 0.039195] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.040657] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.041020] ... version: 2 [ 0.041915] ... bit width: 48 [ 0.042010] ... generic registers: 4 [ 0.042857] ... value mask: 0000ffffffffffff [ 0.043010] ... max period: 00007fffffffffff [ 0.044007] ... fixed-purpose events: 3 [ 0.044822] ... event mask: 000000070000000f [ 0.045280] rcu: Hierarchical SRCU implementation. [ 0.047217] smp: Bringing up secondary CPUs ... [ 0.048403] x86: Booting SMP configuration: [ 0.049019] .... node #0, CPUs: #1 #2 #3 [ 0.055074] smp: Brought up 1 node, 4 CPUs [ 0.056933] smpboot: Max logical packages: 1 [ 0.057020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.086982] node 0 deferred pages initialised in 29ms [ 0.096367] devtmpfs: initialized [ 0.098271] x86/mm: Memory block size: 128MB [ 0.102206] gcov: version magic: 0x41383552 [ 0.104220] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.106056] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.107329] pinctrl core: initialized pinctrl subsystem [ 0.109146] [ 0.109725] ************************************************************* [ 0.112016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.114012] ** ** [ 0.116009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.118011] ** ** [ 0.120013] ** This means that this kernel is built to expose internal ** [ 0.122008] ** IOMMU data structures, which may compromise security on ** [ 0.123006] ** your system. ** [ 0.124010] ** ** [ 0.126130] ** If you see this message and you are not debugging the ** [ 0.128017] ** kernel, report this immediately to your vendor! ** [ 0.130016] ** ** [ 0.132015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.135018] ************************************************************* [ 0.138752] NET: Registered protocol family 16 [ 0.140562] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.143069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.146060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.150020] cpuidle: using governor menu [ 0.152213] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.153407] PCI: Using configuration type 1 for base access [ 0.154093] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.160056] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.162037] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.164074] cryptd: max_cpu_qlen set to 1000 [ 0.165247] ACPI: Added _OSI(Module Device) [ 0.167008] ACPI: Added _OSI(Processor Device) [ 0.168012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.169008] ACPI: Added _OSI(Processor Aggregator Device) [ 0.172172] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.177524] ACPI: Interpreter enabled [ 0.178042] ACPI: PM: (supports S0 S3 S4 S5) [ 0.179022] ACPI: Using IOAPIC for interrupt routing [ 0.180075] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.182340] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.190195] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.192037] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.194011] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.197055] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.201128] acpiphp: Slot [2] registered [ 0.202073] acpiphp: Slot [5] registered [ 0.203071] acpiphp: Slot [6] registered [ 0.204055] acpiphp: Slot [7] registered [ 0.205010] acpiphp: Slot [8] registered [ 0.206005] acpiphp: Slot [9] registered [ 0.206992] acpiphp: Slot [10] registered [ 0.208069] acpiphp: Slot [3] registered [ 0.209071] acpiphp: Slot [4] registered [ 0.210058] acpiphp: Slot [11] registered [ 0.211037] acpiphp: Slot [12] registered [ 0.212049] acpiphp: Slot [13] registered [ 0.213083] acpiphp: Slot [14] registered [ 0.214051] acpiphp: Slot [15] registered [ 0.215038] acpiphp: Slot [16] registered [ 0.216049] acpiphp: Slot [17] registered [ 0.217047] acpiphp: Slot [18] registered [ 0.218046] acpiphp: Slot [19] registered [ 0.219164] acpiphp: Slot [20] registered [ 0.220069] acpiphp: Slot [21] registered [ 0.221053] acpiphp: Slot [22] registered [ 0.223095] acpiphp: Slot [23] registered [ 0.225075] acpiphp: Slot [24] registered [ 0.226101] acpiphp: Slot [25] registered [ 0.227024] acpiphp: Slot [26] registered [ 0.227840] acpiphp: Slot [27] registered [ 0.229059] acpiphp: Slot [28] registered [ 0.230065] acpiphp: Slot [29] registered [ 0.231063] acpiphp: Slot [30] registered [ 0.231987] acpiphp: Slot [31] registered [ 0.233043] PCI host bridge to bus 0000:00 [ 0.233997] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.235015] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.237018] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.238009] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.240011] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.241012] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.242142] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.244766] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.247344] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.253766] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.258052] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.261014] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.263013] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.265017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.267576] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.271000] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.273040] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.275829] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.281012] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.291964] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.297015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.304353] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.312017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.321017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.355016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.369659] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.375015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.398029] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.415016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.427562] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.435021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.443023] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.459014] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.468940] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.474021] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.479000] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.490015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.501839] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.519035] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.528028] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.540019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.551600] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.566018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.578024] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.616013] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.629944] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.632373] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.635320] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.637492] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.640221] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.645704] iommu: Default domain type: Passthrough [ 0.647415] SCSI subsystem initialized [ 0.649152] ACPI: bus type USB registered [ 0.651155] usbcore: registered new interface driver usbfs [ 0.653090] usbcore: registered new interface driver hub [ 0.655073] usbcore: registered new device driver usb [ 0.657175] pps_core: LinuxPPS API ver. 1 registered [ 0.659009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.662056] PTP clock support registered [ 0.664082] EDAC MC: Ver: 3.0.0 [ 0.665133] PCI: Using ACPI for IRQ routing [ 0.667857] NetLabel: Initializing [ 0.668006] NetLabel: domain hash size = 128 [ 0.671011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.672076] NetLabel: unlabeled traffic allowed by default [ 0.674195] vgaarb: loaded [ 0.676349] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.678014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.683506] clocksource: Switched to clocksource kvm-clock [ 0.800371] VFS: Disk quotas dquot_6.6.0 [ 0.801818] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.804816] *** VALIDATE ramfs *** [ 0.806141] *** VALIDATE hugetlbfs *** [ 0.807680] pnp: PnP ACPI init [ 0.809403] pnp: PnP ACPI: found 6 devices [ 0.823278] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.825420] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.827181] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.829653] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.831083] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.833304] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.836314] NET: Registered protocol family 2 [ 0.838943] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.842648] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.846072] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.851674] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.855138] TCP: Hash tables configured (established 65536 bind 65536) [ 0.858126] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.861137] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.864292] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.867104] NET: Registered protocol family 1 [ 0.870199] RPC: Registered named UNIX socket transport module. [ 0.872252] RPC: Registered udp transport module. [ 0.874334] RPC: Registered tcp transport module. [ 0.875943] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.878327] NET: Registered protocol family 44 [ 0.879756] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.882153] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.884146] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.886585] PCI: CLS 0 bytes, default 64 [ 0.888469] Unpacking initramfs... [ 2.445279] debug: unmapping init [mem 0xffff8b99fcc54000-0xffff8b99fffbffff] [ 2.451691] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.454070] software IO TLB: mapped [mem 0x00000000a3800000-0x00000000a7800000] (64MB) [ 2.458242] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.977278] Initialise system trusted keyrings [ 2.979402] Key type blacklist registered [ 2.981453] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.992799] zbud: loaded [ 2.995996] *** VALIDATE nfs *** [ 2.997401] *** VALIDATE nfs4 *** [ 2.999176] pstore: using deflate compression [ 3.003220] Platform Keyring initialized [ 3.112456] NET: Registered protocol family 38 [ 3.114499] Key type asymmetric registered [ 3.115935] Asymmetric key parser 'x509' registered [ 3.117860] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.121693] io scheduler mq-deadline registered [ 3.123565] io scheduler kyber registered [ 3.126957] io scheduler bfq registered [ 3.129103] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.134195] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.137473] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.140684] ACPI: Power Button [PWRF] [ 3.146194] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.153586] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.191172] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.201052] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.222574] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.250953] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.279717] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.283394] Non-volatile memory driver v1.3 [ 3.284763] Linux agpgart interface v0.103 [ 3.310673] virtio_blk virtio1: [vda] 145808 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.315599] vda: detected capacity change from 0 to 74653696 [ 3.329954] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.332091] vdb: detected capacity change from 0 to 1073741824 [ 3.344464] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.347131] vdc: detected capacity change from 0 to 2621440000 [ 3.360199] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.363461] vdd: detected capacity change from 0 to 2621440000 [ 3.377692] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.380175] vde: detected capacity change from 0 to 4294967296 [ 3.396060] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.399584] vdf: detected capacity change from 0 to 4294967296 [ 3.408678] libphy: Fixed MDIO Bus: probed [ 3.415737] usbcore: registered new interface driver usbserial_generic [ 3.418224] usbserial: USB Serial support registered for generic [ 3.420555] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.424676] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.426415] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.428721] mousedev: PS/2 mouse device common for all mice [ 3.431444] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.433698] rtc_cmos 00:05: RTC can wake from S4 [ 3.438910] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.439223] rtc_cmos 00:05: registered as rtc0 [ 3.443688] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.445782] intel_pstate: CPU model not supported [ 3.447626] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.451442] hid: raw HID events driver (C) Jiri Kosina [ 3.453040] usbcore: registered new interface driver usbhid [ 3.454451] usbhid: USB HID core driver [ 3.455691] drop_monitor: Initializing network drop monitor service [ 3.457508] Initializing XFRM netlink socket [ 3.458974] NET: Registered protocol family 10 [ 3.464916] Segment Routing with IPv6 [ 3.466804] NET: Registered protocol family 17 [ 3.469680] mpls_gso: MPLS GSO support [ 3.475468] RAS: Correctable Errors collector initialized. [ 3.477919] AVX version of gcm_enc/dec engaged. [ 3.479759] AES CTR mode by8 optimization enabled [ 3.575113] sched_clock: Marking stable (3575089202, 0)->(4409976904, -834887702) [ 3.579811] registered taskstats version 1 [ 3.584147] Loading compiled-in X.509 certificates [ 3.585776] zswap: loaded using pool lzo/zbud [ 3.615281] Key type big_key registered [ 3.632777] Key type encrypted registered [ 3.634617] ima: No TPM chip found, activating TPM-bypass! [ 3.637190] ima: Allocated hash algorithm: sha1 [ 3.638989] ima: No architecture policies found [ 3.641190] evm: Initialising EVM extended attributes: [ 3.643313] evm: security.selinux [ 3.644676] evm: security.ima [ 3.645954] evm: security.capability [ 3.647565] evm: HMAC attrs: 0x1 [ 3.650388] rtc_cmos 00:05: setting system clock to 2026-07-20 21:40:40 UTC (1784583640) [ 3.656777] debug: unmapping init [mem 0xffffffff90403000-0xffffffff905fffff] [ 3.660248] debug: unmapping init [mem 0xffffffff8f182000-0xffffffff8f458fff] [ 3.669127] Write protecting the kernel read-only data: 28672k [ 3.674261] debug: unmapping init [mem 0xffffffff8d803000-0xffffffff8d9fffff] [ 3.677440] debug: unmapping init [mem 0xffffffff8e114000-0xffffffff8e1fffff] [ 3.718046] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.726438] systemd[1]: Detected virtualization kvm. [ 3.729382] systemd[1]: Detected architecture x86-64. [ 3.731406] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.762512] systemd[1]: No hostname configured. [ 3.764453] systemd[1]: Set hostname to . [ 3.767419] random: systemd: uninitialized urandom read (16 bytes read) [ 3.770350] systemd[1]: Initializing machine ID from random generator. [ 3.853531] random: ln: uninitialized urandom read (6 bytes read) [ 3.962178] random: systemd: uninitialized urandom read (16 bytes read) [ 3.965147] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.969680] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.979205] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local File Systems. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.725874] device-mapper: uevent: version 1.0.3 [ 4.728214] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.969801] virtio_net virtio0 ens2: renamed from eth0 Starting dracut initqueue hook... [ 6.033469] random: fast init done [ 6.215602] scsi host0: ata_piix [ 6.234412] scsi host1: ata_piix [ 6.237386] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.241805] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 11.199097] random: crng init done [ 11.201299] random: 7 urandom warning(s) missed due to ratelimiting [ 12.480328] dracut-initqueue[569]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 15.592181] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 19.601641] printk: systemd: 26 output lines suppressed due to ratelimiting [ 20.944661] SELinux: Disabled at runtime. [ 21.187493] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 21.230352] systemd[1]: Detected virtualization kvm. [ 21.233130] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 24.014440] systemd[1]: initrd-switch-root.service: Succeeded. [ 24.027507] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 24.053103] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 24.072224] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 24.080884] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 24.112531] systemd[1]: Starting Journal Service... Starting Journal Service... [ 24.132784] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Listening on udev Control Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd Root File System. Starting Remount Root and Kernel File Systems... Mounting Huge Pages File System... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting udev Coldplug all Devices... Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd File Systems. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. [ OK [ 24.798929] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 25.879662] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 27.775765] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 27.841254] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 28.573739] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 28.821816] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit) [ ***] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (12s / no limit) [ *] A start job is running for Configur…only root support (13s / no limit) [ **] A start job is running for Configur…only root support (13s / [ 37.376699] Key type dns_resolver registered no limit) [ ***] A start job is running for Configur…only root support (14s / no limit) [ *** ] A start job is running for Configur…only root support (14s / no limit)[ 39.046278] NFS: Registering the id_resolver key type [ 39.048638] Key type id_resolver registered [ 39.050028] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (15s / no limit) [*** ] A start job is running for Configur…only root support (15s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg159-server login: [ 56.551175] hrtimer: interrupt took 8163820 ns [ 125.200551] libcfs: loading out-of-tree module taints kernel. [ 125.264989] Key type ._llcrypt registered [ 125.267641] Key type .llcrypt registered [ 125.388545] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [ 150.954366] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 153.768430] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 153.804178] alg: No test for adler32 (adler32-zlib) [ 155.416026] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 156.450153] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 158.418611] Key type lgssc registered [ 161.276181] Lustre: Echo OBD driver; http://www.lustre.org/ [ 187.993864] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 241.751191] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 259.135762] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 259.186508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 260.622748] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 260.690723] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 260.895483] Lustre: lustre-MDT0000: new disk, initializing [ 261.095231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 261.131256] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 266.445950] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 283.084623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 283.295289] Lustre: 6507:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 283.344762] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 283.351212] Lustre: Skipped 1 previous similar message [ 283.437610] Lustre: lustre-MDT0001: new disk, initializing [ 283.481992] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 283.515545] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 283.525857] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 288.277956] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 293.462221] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 306.782646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 307.219483] Lustre: lustre-OST0000: new disk, initializing [ 307.223318] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 307.231260] Lustre: 8446:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 307.344189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 309.190182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 309.196703] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 309.320646] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 313.625464] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 328.589119] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 328.706337] Lustre: lustre-OST0001: new disk, initializing [ 328.716321] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 328.732287] Lustre: 9519:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 328.859059] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 335.226484] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 335.971783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 335.989211] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 336.104197] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 347.749233] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 356.534437] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 362.962587] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing check_logdir /tmp/testlogs/ [ 368.141163] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing yml_node [ 373.627274] Lustre: DEBUG MARKER: Client: 2.17.54.183 [ 375.948760] Lustre: DEBUG MARKER: MDS: 2.17.54.183 [ 378.283738] Lustre: DEBUG MARKER: OSS: 2.17.54.183 [ 379.755770] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Jul 20 17:46:55 EDT 2026 [ 401.313083] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 403.961569] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 421.858106] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 421.877333] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 421.904487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 423.394437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 423.394856] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 423.410300] Lustre: Skipped 1 previous similar message [ 423.469069] Lustre: Skipped 1 previous similar message [ 427.774375] Lustre: server umount lustre-MDT0000 complete [ 432.576144] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784584069 with bad export cookie 12510612067818571387 [ 432.583264] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 432.591468] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 433.640846] LustreError: 11083:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 433.642437] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 433.655055] LustreError: 11083:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 433.675530] Lustre: Skipped 1 previous similar message [ 433.681336] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 433.685167] Lustre: Skipped 3 previous similar messages [ 437.666141] LustreError: 11083:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 437.673622] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 437.678977] LustreError: 11083:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 437.706757] Lustre: Skipped 1 previous similar message [ 439.327151] Lustre: server umount lustre-MDT0001 complete [ 443.890112] Lustre: server umount lustre-OST0000 complete [ 449.703331] Lustre: server umount lustre-OST0001 complete [ 459.605878] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [ 471.722821] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 511.828501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 512.033803] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 512.057108] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 512.137663] Lustre: lustre-MDT0000: new disk, initializing [ 512.233785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 512.251839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 516.814189] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 526.144461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 526.251405] Lustre: 15081:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 526.301908] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 526.305382] Lustre: Skipped 1 previous similar message [ 526.358297] Lustre: lustre-MDT0001: new disk, initializing [ 526.475506] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 526.540537] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 526.548728] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 531.004417] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 540.189261] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 545.344670] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 551.544607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 551.944471] Lustre: lustre-OST0000: new disk, initializing [ 551.948498] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 551.954286] Lustre: 17050:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 552.018301] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 553.133970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 553.140869] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 553.293503] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 558.366719] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 567.277366] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 572.412902] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 572.628515] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 577.275942] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 577.544683] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 584.162450] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 584.174562] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 584.188118] Lustre: Skipped 1 previous similar message [ 584.198348] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 584.207029] Lustre: Skipped 1 previous similar message [ 585.786907] Lustre: server umount lustre-OST0000 complete [ 589.282040] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 589.290301] Lustre: Skipped 2 previous similar messages [ 594.402148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 594.413793] Lustre: Skipped 3 previous similar messages [ 595.654659] Lustre: server umount lustre-MDT0000 complete [ 599.211816] LustreError: 15073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784584236 with bad export cookie 12510612067818572920 [ 599.214445] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 599.229984] LustreError: 15073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 599.593568] Lustre: server umount lustre-MDT0001 complete [ 605.805499] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 17:50:40 (1784584240) [ 624.102611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 624.810764] LustreError: 19926:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 624.912950] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 629.959241] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 630.244245] LustreError: 19927:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 634.347553] LustreError: 19926:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 638.788456] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 644.503754] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 652.898195] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 658.494756] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 661.086477] Lustre: Failing over lustre-MDT0000 [ 661.452130] Lustre: server umount lustre-MDT0000 complete [ 664.168529] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 664.191449] LustreError: 19922:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 664.219173] LustreError: 19922:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 680.929767] Lustre: 3637:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784584301/real 1784584301] req@ffff8b9a73e97b80 x1871271735491712/t0(0) o400->MGC192.168.201.159@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1784584317 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 680.952938] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 680.967413] LustreError: 19926:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 680.984931] LustreError: 19926:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 681.826921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 691.544605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 691.554044] Lustre: Skipped 1 previous similar message [ 691.629438] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 696.554595] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 696.820498] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 696.821876] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 696.868058] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 704.761632] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 709.543871] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 709.548837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 709.557132] Lustre: Skipped 2 previous similar messages [ 709.574831] Lustre: Skipped 1 previous similar message [ 714.906230] Lustre: server umount lustre-MDT0000 complete [ 719.004386] LustreError: 21495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784584355 with bad export cookie 12510612067818574334 [ 719.017063] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 719.027902] LustreError: 21495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 719.239518] Lustre: server umount lustre-MDT0001 complete [ 728.263265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 728.842049] LustreError: 23267:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 728.857060] LustreError: 23267:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 734.084684] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 743.127547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 743.393890] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 743.400730] Lustre: Skipped 2 previous similar messages [ 748.462937] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 756.438150] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 762.731532] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 765.174312] Lustre: Failing over lustre-MDT0000 [ 765.407910] Lustre: server umount lustre-MDT0000 complete [ 768.993823] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 769.006403] LustreError: Skipped 1 previous similar message [ 769.015645] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 785.098095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 785.174142] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 785.423262] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 790.503851] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 790.526147] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 790.541717] Lustre: Skipped 1 previous similar message [ 790.560648] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 790.858812] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 800.696894] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 805.859510] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 810.978512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 810.983100] Lustre: Skipped 3 previous similar messages [ 819.168161] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 819.307917] Lustre: server umount lustre-MDT0000 complete [ 821.218329] LustreError: 23267:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 821.237717] LustreError: 23267:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 822.986340] LustreError: 23249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784584459 with bad export cookie 12510612067818575272 [ 823.008454] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 823.348402] Lustre: server umount lustre-MDT0001 complete [ 832.676246] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 17:54:27 (1784584467) [ 851.345153] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [ 861.937548] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 900.027530] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 909.958721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 910.186338] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 910.215123] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 910.308797] Lustre: lustre-MDT0000: new disk, initializing [ 910.435195] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 910.446148] Lustre: Skipped 3 previous similar messages [ 910.496957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 915.087336] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 925.288426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 925.412683] Lustre: 29645:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 925.671668] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 929.983393] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 939.266558] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 944.770900] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 951.159102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 951.555457] Lustre: 31612:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 952.777229] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 952.818299] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 952.950299] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 952.963592] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 958.213547] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 967.155801] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 972.401014] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 993.766417] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 993.767433] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 993.776823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 993.776832] Lustre: Skipped 2 previous similar messages [ 993.797055] Lustre: Skipped 4 previous similar messages [ 993.830042] LustreError: Skipped 1 previous similar message [ 996.683821] Lustre: server umount lustre-OST0000 complete [ 1002.474807] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1006.945694] Lustre: server umount lustre-MDT0000 complete [ 1007.010281] LustreError: 30807:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1007.043466] LustreError: 30807:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1010.539641] LustreError: 29635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784584647 with bad export cookie 12510612067818575678 [ 1010.542464] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1010.553098] LustreError: 29635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 1011.310252] Lustre: server umount lustre-MDT0001 complete [ 1023.809847] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 1027.118878] Key type lgssc unregistered [ 1027.464494] LNet: 33589:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1027.474604] LNetError: 33589:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1027.495366] LNet: Removed LNI 192.168.201.159@tcp [ 1028.441200] Key type .llcrypt unregistered [ 1028.445689] Key type ._llcrypt unregistered [ 1053.551714] Key type ._llcrypt registered [ 1053.554384] Key type .llcrypt registered [ 1053.641645] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 1055.015503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1055.041655] alg: No test for adler32 (adler32-zlib) [ 1056.082210] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 1056.323184] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 1057.984200] Key type lgssc registered [ 1058.989626] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1066.306527] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1066.340439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1067.669864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1072.258579] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1077.218248] LustreError: 35001:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1077.244118] LustreError: 35001:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1080.047101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1080.343109] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1085.239657] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1093.651255] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1098.376678] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1104.221700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1104.496066] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1105.525099] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1109.973933] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1112.727617] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1118.111921] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1131.186231] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 17:59:26 (1784584766) [ 1133.195379] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1135.320961] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1143.984492] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 17:59:38 (1784584778) [ 1152.432746] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 17:59:47 (1784584787) [ 1153.767489] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1162.325633] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1162.336961] Lustre: Skipped 2 previous similar messages [ 1164.089810] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 17:59:59 (1784584799) [ 1167.377384] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1167.381553] Lustre: Skipped 3 previous similar messages [ 1184.287472] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1184.293866] Lustre: Skipped 6 previous similar messages [ 1187.514835] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 18:00:22 (1784584822) [ 1190.749712] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1195.218550] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1205.542438] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 18:00:40 (1784584840) [ 1207.323529] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1207.328854] Lustre: Skipped 3 previous similar messages [ 1221.088174] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 18:00:56 (1784584856) [ 1235.938145] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1235.946672] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1235.954816] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1235.975441] Lustre: Skipped 1 previous similar message [ 1241.057448] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1241.070037] Lustre: Skipped 2 previous similar messages [ 1246.179130] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1249.248118] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1249.549061] Lustre: server umount lustre-OST0000 complete [ 1254.369549] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1254.376808] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1254.391291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1254.400266] Lustre: Skipped 1 previous similar message [ 1256.421131] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1261.540543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1261.550678] Lustre: Skipped 3 previous similar messages [ 1268.192244] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1268.340547] Lustre: server umount lustre-MDT0000 complete [ 1271.785489] LustreError: 35001:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1271.827766] LustreError: 35001:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1273.137450] LustreError: 39424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784584909 with bad export cookie 16480143195860653700 [ 1273.151715] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1273.153444] LustreError: 39424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1273.748849] Lustre: server umount lustre-MDT0001 complete [ 1286.045573] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 1288.634705] Key type lgssc unregistered [ 1288.926985] LNet: 42875:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1288.945482] LNetError: 42875:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1288.967250] LNet: Removed LNI 192.168.201.159@tcp [ 1289.954877] Key type .llcrypt unregistered [ 1289.961068] Key type ._llcrypt unregistered [ 1310.681321] Key type ._llcrypt registered [ 1310.683695] Key type .llcrypt registered [ 1310.870264] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [ 1327.470649] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 1329.076859] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1329.213223] alg: No test for adler32 (adler32-zlib) [ 1330.319947] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 1330.712310] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 1332.376743] Key type lgssc registered [ 1333.388512] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1366.043681] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1366.070899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1367.300579] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1367.324402] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1367.381890] Lustre: lustre-MDT0000: new disk, initializing [ 1367.441584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1367.453994] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1371.199743] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1381.743616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1381.904509] Lustre: 46034:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1382.004742] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1382.010966] Lustre: Skipped 1 previous similar message [ 1382.149204] Lustre: lustre-MDT0001: new disk, initializing [ 1382.286538] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1382.331533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1382.361302] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1386.691776] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1396.759848] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1403.042772] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1409.986251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1410.201223] Lustre: lustre-OST0000: new disk, initializing [ 1410.204774] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1410.210538] Lustre: 48001:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1410.282714] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1416.380428] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1419.373975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1419.390209] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1419.568394] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1427.263458] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1432.216479] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1432.420941] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1436.859423] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1437.087148] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1440.226080] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1440.238513] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1440.257285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1440.263673] Lustre: Skipped 1 previous similar message [ 1440.276232] Lustre: Skipped 1 previous similar message [ 1445.163557] Lustre: server umount lustre-OST0000 complete [ 1450.469343] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1450.472483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1450.481694] Lustre: Skipped 1 previous similar message [ 1450.493648] Lustre: Skipped 1 previous similar message [ 1455.506360] Lustre: server umount lustre-MDT0000 complete [ 1459.294097] LustreError: 46026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784585096 with bad export cookie 5868137966978330215 [ 1459.300884] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1459.311564] LustreError: 46026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1459.826420] Lustre: server umount lustre-MDT0001 complete [ 1470.508900] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 18:05:05 (1784585105) [ 1477.717673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1478.188920] LustreError: 50460:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1478.208941] LustreError: 50460:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1478.290318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1482.741887] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1483.749290] LustreError: 50461:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1488.869244] LustreError: 50460:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1492.532089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1492.904102] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1498.523253] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1506.167403] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1511.882392] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1519.089861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1519.507116] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1526.497935] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1534.054477] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1542.010625] LustreError: 52988:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1543.223329] Lustre: Modifying parameter general.timeout=1 in log params [ 1544.287408] Lustre: Modifying parameter general.timeout=2 in log params [ 1547.220520] Lustre: Modifying parameter general.timeout=5 in log params [ 1547.227689] Lustre: Skipped 2 previous similar messages [ 1551.650408] Lustre: Modifying parameter general.timeout=9 in log params [ 1551.653855] Lustre: Skipped 3 previous similar messages [ 1560.274134] Lustre: Modifying parameter general.timeout=19 in log params [ 1560.284060] Lustre: Skipped 9 previous similar messages [ 1576.584984] Lustre: Modifying parameter general.timeout=34 in log params [ 1576.596441] Lustre: Skipped 14 previous similar messages [ 1608.799730] Lustre: Modifying parameter general.timeout=68 in log params [ 1608.803228] Lustre: Skipped 33 previous similar messages [ 1677.229439] Lustre: Disabling parameter general.timeout= in log params [ 1677.232755] Lustre: Skipped 61 previous similar messages [ 1680.770824] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 18:08:35 (1784585315) [ 1684.412876] Lustre: 59939:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1687.047659] Lustre: 60038:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1689.425813] Lustre: 60147:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1695.382134] Lustre: 60445:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1695.389043] Lustre: 60445:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1724.049774] Lustre: 61806:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1724.070189] LustreError: 61806:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1727.491850] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 18:09:22 (1784585362) [ 1740.260471] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1740.264990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1740.304944] Lustre: Skipped 2 previous similar messages [ 1740.338872] Lustre: Skipped 2 previous similar messages [ 1745.381092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1745.388654] Lustre: Skipped 2 previous similar messages [ 1745.977927] Lustre: server umount lustre-MDT0000 complete [ 1749.209733] LustreError: 50442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784585386 with bad export cookie 5868137966978331097 [ 1749.212757] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1749.226907] LustreError: 50442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 1749.700312] Lustre: server umount lustre-MDT0001 complete [ 1763.654317] Lustre: server umount lustre-OST0000 complete [ 1795.176748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1795.638387] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1796.090373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1801.121388] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1810.944195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1811.055726] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1811.106824] Lustre: 64340:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1816.191604] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1824.349530] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1829.723709] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1835.990589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1836.152039] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1836.351430] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1836.356019] Lustre: Skipped 1 previous similar message [ 1843.143870] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1850.042507] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1851.580382] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1851.588661] Lustre: Skipped 23 previous similar messages [ 1851.627130] Lustre: 66907:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1862.629143] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1862.643599] LustreError: Skipped 1 previous similar message [ 1862.653784] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1862.681151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1865.636643] Lustre: server umount lustre-OST0000 complete [ 1872.865624] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1872.869269] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1872.885200] LustreError: Skipped 1 previous similar message [ 1872.890672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1872.905215] Lustre: Skipped 2 previous similar messages [ 1872.929821] Lustre: Skipped 2 previous similar messages [ 1876.350410] Lustre: server umount lustre-MDT0000 complete [ 1877.986026] LustreError: 64346:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1877.997447] LustreError: 64346:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1880.324496] LustreError: 66202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784585517 with bad export cookie 5868137966978334163 [ 1880.329953] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1880.337632] LustreError: 66202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1880.697840] Lustre: server umount lustre-MDT0001 complete [ 1891.261066] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 1894.006294] Key type lgssc unregistered [ 1894.340845] LNet: 68113:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1894.347645] LNetError: 68113:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1894.366989] LNet: Removed LNI 192.168.201.159@tcp [ 1895.239412] Key type .llcrypt unregistered [ 1895.240852] Key type ._llcrypt unregistered [ 1923.547952] Key type ._llcrypt registered [ 1923.550784] Key type .llcrypt registered [ 1923.657622] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 1924.780262] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1925.025852] alg: No test for adler32 (adler32-zlib) [ 1926.400921] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 1926.851134] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 1928.624173] Key type lgssc registered [ 1930.521410] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1941.417937] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1941.470282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1943.343762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1949.378680] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1953.765469] LustreError: 69528:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1953.804290] LustreError: 69528:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1958.882732] LustreError: 69529:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1959.082526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1959.760869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1964.594660] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 1972.202694] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1978.045306] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1986.566530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1987.251684] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1994.056797] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2002.743927] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2013.941214] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2025.953743] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2025.964564] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2025.981192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2026.473727] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2026.482219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2026.491874] Lustre: Skipped 1 previous similar message [ 2026.504812] Lustre: Skipped 2 previous similar messages [ 2028.670033] Lustre: server umount lustre-MDT0000 complete [ 2031.588278] LustreError: 70262:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2031.613983] LustreError: 70262:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2032.548027] LustreError: 69508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784585669 with bad export cookie 7471404420561826868 [ 2032.573898] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2033.149216] Lustre: server umount lustre-MDT0001 complete [ 2046.491613] Lustre: server umount lustre-OST0000 complete [ 2076.786400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2077.008143] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2077.072217] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2077.341612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2082.104549] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2090.792734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2090.877897] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2090.910373] Lustre: 74402:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2091.429657] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2095.571935] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2103.393154] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2108.759693] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2117.527303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2117.824559] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2118.477605] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2125.587708] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2132.393083] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2134.146642] Lustre: 76980:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2135.787903] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2144.746747] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2144.746813] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2144.761881] LustreError: Skipped 1 previous similar message [ 2144.771041] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2144.771623] Lustre: Skipped 1 previous similar message [ 2144.785703] Lustre: Skipped 1 previous similar message [ 2146.732985] Lustre: server umount lustre-OST0000 complete [ 2152.929649] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2152.940176] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2152.987327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2156.965371] Lustre: server umount lustre-MDT0000 complete [ 2160.097427] LustreError: 75550:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2160.110938] LustreError: 75550:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2160.656210] LustreError: 74396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784585797 with bad export cookie 7471404420561827715 [ 2160.657943] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2160.669492] LustreError: 74396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2161.115515] Lustre: server umount lustre-MDT0001 complete [ 2173.022158] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 2175.657574] Key type lgssc unregistered [ 2176.034033] LNet: 78189:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2176.043956] LNetError: 78189:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2176.067617] LNet: Removed LNI 192.168.201.159@tcp [ 2177.095244] Key type .llcrypt unregistered [ 2177.098507] Key type ._llcrypt unregistered [ 2200.884952] Key type ._llcrypt registered [ 2200.887957] Key type .llcrypt registered [ 2200.992328] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 2202.576288] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2202.627028] alg: No test for adler32 (adler32-zlib) [ 2203.854943] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 2204.159945] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 2205.792183] Key type lgssc registered [ 2206.920973] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2215.540362] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2215.551589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2217.004689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2222.378328] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2227.174024] LustreError: 79601:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2227.196543] LustreError: 79601:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2231.933118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2232.342623] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2236.897528] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2244.369109] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2250.669589] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2257.745837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2258.444135] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2264.805802] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2272.346414] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2282.815679] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2283.841939] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2285.756890] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2285.762460] Lustre: Skipped 1 previous similar message [ 2288.125833] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2288.132349] Lustre: Skipped 1 previous similar message [ 2292.258234] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2292.268288] Lustre: Skipped 3 previous similar messages [ 2300.393490] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2300.399153] Lustre: Skipped 6 previous similar messages [ 2316.465903] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2316.477771] Lustre: Skipped 15 previous similar messages [ 2348.619406] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2348.623422] Lustre: Skipped 35 previous similar messages [ 2382.556720] Lustre: 87704:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2385.398733] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 18:20:20 (1784586020) [ 2387.188259] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2389.037817] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 18:20:24 (1784586024) [ 2394.601657] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2394.606407] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2394.612304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2394.638051] Lustre: Skipped 1 previous similar message [ 2398.660029] Lustre: server umount lustre-OST0000 complete [ 2404.833486] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2404.860330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2404.866618] Lustre: Skipped 2 previous similar messages [ 2409.953340] LustreError: 79596:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2409.964392] LustreError: 79596:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2410.777197] Lustre: server umount lustre-MDT0000 complete [ 2414.165183] LustreError: 79581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784586051 with bad export cookie 16977488483321779096 [ 2414.167053] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2414.180941] LustreError: 79581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2414.574177] Lustre: server umount lustre-MDT0001 complete [ 2425.607178] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 2428.306550] Key type lgssc unregistered [ 2428.650262] LNet: 88951:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2428.661096] LNetError: 88951:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2429.734537] LNet: Removed LNI 192.168.201.159@tcp [ 2430.886501] Key type .llcrypt unregistered [ 2430.888538] Key type ._llcrypt unregistered [ 2435.647401] Key type ._llcrypt registered [ 2435.650963] Key type .llcrypt registered [ 2435.799683] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2440.109084] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules [ 2441.164157] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2441.186694] alg: No test for adler32 (adler32-zlib) [ 2442.419200] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 2442.620191] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 2444.264247] Key type lgssc registered [ 2445.515811] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2446.830908] LustreError: 90154:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2463.735717] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 2468.837833] LustreError: 90154:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2475.895438] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2475.922871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2477.633933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2483.237269] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2487.782830] LustreError: 91177:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2487.816581] LustreError: 91177:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2492.898454] LustreError: 91178:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2494.113891] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 18:22:08 (1784586128) [ 2498.019106] LustreError: 91177:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2499.153167] LustreError: 91209:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2499.165400] LustreError: 91209:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2499.171630] LustreError: 91209:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 22, retries 0, failed: rc = -5 [ 2499.519826] Lustre: server umount lustre-MDT0000 complete [ 2513.283509] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 2516.330326] Key type lgssc unregistered [ 2516.584152] LNet: 92703:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2516.594681] LNetError: 92703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2516.626270] LNet: Removed LNI 192.168.201.159@tcp [ 2517.631877] Key type .llcrypt unregistered [ 2517.634132] Key type ._llcrypt unregistered [ 2536.374570] Key type ._llcrypt registered [ 2536.377301] Key type .llcrypt registered [ 2536.573575] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 2538.061490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2538.171980] alg: No test for adler32 (adler32-zlib) [ 2539.175371] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 2539.489310] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 2541.192178] Key type lgssc registered [ 2542.833474] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2551.597588] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2551.614678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2553.234593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2558.229583] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2562.532979] LustreError: 93921:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2562.560975] LustreError: 93921:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2567.305850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2567.658114] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2568.179003] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2572.979654] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2581.389758] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2587.346218] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2594.170112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2594.741116] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2600.911074] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2608.667418] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2663.245965] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 18:24:58 (1784586298) [ 2664.697539] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2666.971261] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 18:25:01 (1784586301) [ 2670.562496] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2670.577120] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2670.602724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2674.662758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2674.670588] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2674.692611] Lustre: Skipped 1 previous similar message [ 2676.816280] Lustre: server umount lustre-MDT0000 complete [ 2679.784446] LustreError: 93921:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2679.809862] LustreError: 93921:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2680.541277] LustreError: 93901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784586317 with bad export cookie 14043650998483367112 [ 2680.544528] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2680.551908] LustreError: 93901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2681.031742] Lustre: server umount lustre-MDT0001 complete [ 2695.263411] Lustre: server umount lustre-OST0000 complete [ 2707.742107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2708.290313] LustreError: 98201:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2708.569276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2713.360207] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2713.578195] LustreError: 98202:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2718.689631] LustreError: 98201:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2721.336208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2722.037648] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2726.258951] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2734.860374] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2740.470396] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2752.503822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2752.695676] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2752.703788] LustreError: 98189:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2752.722733] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2752.732088] LustreError: 100292:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2752.737255] LustreError: 100292:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2752.740469] LustreError: 100292:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2752.786434] Lustre: server umount lustre-OST0000 complete [ 2752.788593] LustreError: 100292:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2759.994394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2760.125565] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2760.130507] LustreError: 98191:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2760.136478] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2760.141496] LustreError: 100755:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2760.145736] LustreError: 100755:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2760.149834] LustreError: 100755:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2760.209342] Lustre: server umount lustre-OST0000 complete [ 2760.213940] LustreError: 100755:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2769.120642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2769.335978] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2769.339516] Lustre: Found index 0 for lustre-OST0000, updating log [ 2769.346788] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2769.432786] Lustre: lustre-OST0000: new disk, initializing [ 2769.436462] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2769.446588] Lustre: 101290:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2769.739931] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2776.176730] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2779.830756] Lustre: Failing over lustre-OST0000 [ 2779.992941] Lustre: server umount lustre-OST0000 complete [ 2784.226519] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2784.231741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2784.246772] Lustre: Skipped 1 previous similar message [ 2784.258552] Lustre: Skipped 3 previous similar messages [ 2789.346190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2789.356176] Lustre: Skipped 1 previous similar message [ 2793.546810] Lustre: server umount lustre-MDT0000 complete [ 2796.514658] LustreError: 98196:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2796.535055] LustreError: 98196:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2797.404579] LustreError: 98181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784586434 with bad export cookie 14043650998483369856 [ 2797.411497] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2797.417530] LustreError: 98181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2797.860794] Lustre: server umount lustre-MDT0001 complete [ 2807.460471] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 18:27:22 (1784586442) [ 2825.159285] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 2834.697341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2834.942798] LustreError: 103637:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2835.048620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2840.162331] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2850.293340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2855.148767] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2858.462829] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2865.549648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2866.274466] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2872.883562] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2875.495140] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2875.510090] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2875.534629] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2879.587316] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:05 [ 2879.604174] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2879.617264] Lustre: lustre-OST0000: Recovery over after 0:04, of 1 clients 1 recovered and 0 were evicted. [ 2879.618557] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2879.627766] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2880.852303] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2880.936813] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2880.994454] Lustre: lustre-OST0001: new disk, initializing [ 2880.996546] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2881.001367] Lustre: 105955:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2881.245564] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2881.248233] Lustre: Skipped 2 previous similar messages [ 2887.954844] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2888.882942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2888.899605] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2889.060415] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2896.998864] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2903.287899] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2909.672671] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2909.674357] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2909.688458] Lustre: Skipped 2 previous similar messages [ 2909.707078] Lustre: Skipped 4 previous similar messages [ 2914.486206] Lustre: server umount lustre-MDT0001 complete [ 2914.791193] LustreError: 104846:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2914.817686] LustreError: 104846:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 2925.850399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2926.040797] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2926.050303] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2926.119891] Lustre: 103625:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2927.636300] LustreError: 107492:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2927.645882] LustreError: 107492:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2927.668104] LustreError: 103630:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2927.787714] LustreError: 107503:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.159@tcp: cfg command failed: rc = -17 [ 2927.800860] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2927.800860] [ 2932.088900] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 2932.199799] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2941.392333] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 18:29:36 (1784586576) [ 2957.574822] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 2974.588629] Lustre: 109568:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2996.846168] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3000.137249] Lustre: 110704:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3080.174292] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3080.179356] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3080.215757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3080.226078] Lustre: Skipped 1 previous similar message [ 3086.035238] Lustre: server umount lustre-MDT0000 complete [ 3090.647556] LustreError: 107446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784586727 with bad export cookie 14043650998483370780 [ 3090.651143] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3090.664109] LustreError: 107446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3090.923412] LustreError: 103632:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3090.923871] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3090.937625] LustreError: 103632:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 3090.963288] Lustre: Skipped 3 previous similar messages [ 3091.094206] Lustre: server umount lustre-MDT0001 complete [ 3096.387499] Lustre: server umount lustre-OST0000 complete [ 3101.568797] Lustre: server umount lustre-OST0001 complete [ 3108.153153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3117.989927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3126.640351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3136.509809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3155.544191] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 3166.033318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3166.058305] Lustre: lustre-MDT0000: reset Object Index mappings [ 3166.882172] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3166.893977] Lustre: Skipped 1 previous similar message [ 3172.222852] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3181.492824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3181.559530] Lustre: lustre-MDT0001: reset Object Index mappings [ 3187.842474] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3190.882568] Lustre: 115797:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3198.261477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3206.090667] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3206.407947] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 3211.526286] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:382 to 0x280000401:418) [ 3215.304353] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3220.997703] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3221.010342] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:199 to 0x2c0000400:225) [ 3222.534259] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3230.576473] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3234.701682] Lustre: 117673:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3284.959794] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 18:35:19 (1784586919) [ 3313.125669] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3313.146274] Lustre: Skipped 1 previous similar message [ 3313.148917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3313.172543] Lustre: Skipped 10 previous similar messages [ 3316.938389] Lustre: server umount lustre-MDT0000 complete [ 3318.254457] LustreError: 114644:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3318.293509] LustreError: 114644:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 3322.148157] LustreError: 114630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784586958 with bad export cookie 14043650998483417862 [ 3322.161461] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3328.916298] Lustre: server umount lustre-MDT0001 complete [ 3340.763656] Lustre: server umount lustre-OST0000 complete [ 3351.636208] Lustre: server umount lustre-OST0001 complete [ 3359.289957] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [ 3369.679572] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 3409.253738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3409.509685] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3409.540605] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3409.623238] Lustre: lustre-MDT0000: new disk, initializing [ 3409.681537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3409.687124] Lustre: Skipped 3 previous similar messages [ 3409.700256] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3414.865880] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3421.461414] Lustre: server umount lustre-MDT0000 complete [ 3430.815121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3431.126273] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3431.142291] Lustre: 121944:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3431.162215] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3435.996828] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3440.693902] Lustre: server umount lustre-MDT0000 complete [ 3450.253680] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 18:38:05 (1784587085) [ 3451.873094] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3454.487851] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 18:38:08 (1784587088) [ 3473.466514] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 3483.556436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3488.577842] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3496.903789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3497.121061] Lustre: 123992:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3497.210847] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3497.215758] Lustre: Skipped 1 previous similar message [ 3497.356349] Lustre: lustre-MDT0001: new disk, initializing [ 3497.607193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3497.650088] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3502.499381] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3507.492862] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3514.432952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3514.691688] Lustre: lustre-OST0000: new disk, initializing [ 3514.693894] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3514.697206] Lustre: 125569:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3516.722169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3516.736834] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3516.914503] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3520.882177] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3532.150575] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3532.349921] Lustre: lustre-OST0001: new disk, initializing [ 3532.352896] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3532.357979] Lustre: 126437:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3534.089615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3534.109765] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3534.194260] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3538.385406] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 3548.326360] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3552.586329] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3918.289552] Lustre: 132847:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3918.299776] Lustre: 132847:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3918.313751] Lustre: 132847:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3918.323267] Lustre: 132847:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3918.337345] Lustre: 132847:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3918.342591] Lustre: 132847:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4189.373565] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 18:50:23 (1784587823) [ 4205.032348] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4205.081582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4205.081591] Lustre: Skipped 4 previous similar messages [ 4205.100331] Lustre: Skipped 4 previous similar messages [ 4215.270147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4215.278113] Lustre: Skipped 10 previous similar messages [ 4219.360156] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4219.760768] Lustre: server umount lustre-MDT0000 complete [ 4220.393825] LustreError: 124003:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4220.416716] LustreError: 124003:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 4223.069025] LustreError: 123985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784587859 with bad export cookie 14043650998483484831 [ 4223.071291] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4223.086942] LustreError: 123985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4225.510699] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4225.539083] Lustre: Skipped 3 previous similar messages [ 4229.889940] Lustre: server umount lustre-MDT0001 complete [ 4248.032146] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4248.142400] Lustre: server umount lustre-OST0000 complete [ 4258.218665] Lustre: server umount lustre-OST0001 complete [ 4266.261202] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [ 4276.792868] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 4316.240288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4316.678209] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4316.748550] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4316.854419] Lustre: lustre-MDT0000: new disk, initializing [ 4316.925395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4316.927472] Lustre: Skipped 5 previous similar messages [ 4316.939919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4321.543583] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 4334.426655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4334.643387] Lustre: 135909:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4334.752967] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4334.758182] Lustre: Skipped 1 previous similar message [ 4334.972290] Lustre: lustre-MDT0001: new disk, initializing [ 4335.359256] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4335.386974] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4340.364375] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 4351.612418] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4359.083560] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4367.249663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4367.589308] Lustre: lustre-OST0000: new disk, initializing [ 4367.593231] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4367.601421] Lustre: 137877:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4367.669354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4367.677135] Lustre: Skipped 1 previous similar message [ 4369.287155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4369.302465] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4369.426410] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4375.372032] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 4386.292470] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4398.863313] Lustre: lustre-MDD0000: changelog on [ 4402.134371] Lustre: lustre-MDD0001: changelog on [ 6063.103732] Lustre: lustre-MDD0001: changelog off [ 6065.542901] Lustre: 135914:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6066.709371] Lustre: lustre-MDD0000: changelog off [ 6074.355296] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6074.376556] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6074.401349] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6074.410245] Lustre: Skipped 5 previous similar messages [ 6077.421583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6077.432225] Lustre: Skipped 1 previous similar message [ 6082.531403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6082.538087] Lustre: Skipped 1 previous similar message [ 6088.161136] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6088.281760] Lustre: server umount lustre-OST0000 complete [ 6092.776659] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6092.794691] Lustre: Skipped 1 previous similar message [ 6092.804116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6092.809387] Lustre: Skipped 3 previous similar messages [ 6098.981750] Lustre: server umount lustre-MDT0000 complete [ 6102.679750] LustreError: 135899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784589739 with bad export cookie 14043650998484306666 [ 6102.681598] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6102.688077] LustreError: 135899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6102.972851] Lustre: server umount lustre-MDT0001 complete [ 6113.906899] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 6116.656750] Key type lgssc unregistered [ 6116.938081] LNet: 146994:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6116.943382] LNetError: 146994:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6116.953844] LNet: Removed LNI 192.168.201.159@tcp [ 6117.697272] Key type .llcrypt unregistered [ 6117.699686] Key type ._llcrypt unregistered [ 6137.101772] Key type ._llcrypt registered [ 6137.105606] Key type .llcrypt registered [ 6137.193525] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [ 6153.126975] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 6154.325825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6154.428634] alg: No test for adler32 (adler32-zlib) [ 6155.595386] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 6155.824708] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 6157.504182] Key type lgssc registered [ 6158.925426] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6196.743867] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6196.789604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6198.144425] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6198.216080] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6198.336751] Lustre: lustre-MDT0000: new disk, initializing [ 6198.448338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6198.482266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6204.248773] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 6219.552544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6219.741660] Lustre: 150194:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6219.831639] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6219.842252] Lustre: Skipped 1 previous similar message [ 6219.919566] Lustre: lustre-MDT0001: new disk, initializing [ 6220.032755] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6220.095551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6220.115883] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6226.293577] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 6236.521395] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6242.117247] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6250.203150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6250.641755] Lustre: lustre-OST0000: new disk, initializing [ 6250.646351] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6250.652104] Lustre: 152166:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6250.800734] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6256.697152] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6256.718118] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6256.898372] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6260.456250] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 6270.674474] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6276.821741] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_module obdecho/obdecho [ 6279.025532] LustreError: 153170:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6279.035641] LustreError: 153170:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6281.209038] LustreError: 153267:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6292.974444] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6292.979334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6292.986274] Lustre: Skipped 1 previous similar message [ 6298.082753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6298.096968] Lustre: Skipped 4 previous similar messages [ 6298.536453] Lustre: server umount lustre-MDT0000 complete [ 6302.083109] LustreError: 150187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784589938 with bad export cookie 11166410052983977209 [ 6302.099126] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6302.106053] LustreError: 150187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6302.378807] Lustre: server umount lustre-MDT0001 complete [ 6315.486403] Lustre: server umount lustre-OST0000 complete [ 6337.195568] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [ 6347.208164] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 6383.091431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6383.294640] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6383.307349] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6383.365706] Lustre: lustre-MDT0000: new disk, initializing [ 6383.420220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6383.437878] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6388.412678] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 6398.375053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6398.494779] Lustre: 156982:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6398.542855] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6398.545728] Lustre: Skipped 1 previous similar message [ 6398.595607] Lustre: lustre-MDT0001: new disk, initializing [ 6398.708606] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6398.760533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6398.769599] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6403.568391] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 6413.036747] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6418.806737] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6426.539403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6426.815390] Lustre: lustre-OST0000: new disk, initializing [ 6426.819765] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6426.824409] Lustre: 158950:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6426.906966] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6432.881935] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 6436.521953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6436.537608] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6436.654708] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6442.128876] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6446.390844] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6446.612701] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6451.538342] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6451.768682] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6457.314127] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6457.314273] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6457.320798] LustreError: Skipped 1 previous similar message [ 6457.336218] Lustre: Skipped 1 previous similar message [ 6457.338970] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6459.805269] Lustre: server umount lustre-OST0000 complete [ 6465.505343] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6465.512927] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6465.529423] Lustre: Skipped 1 previous similar message [ 6465.534487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6465.546255] Lustre: Skipped 1 previous similar message [ 6467.566580] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6470.059461] Lustre: server umount lustre-MDT0000 complete [ 6473.621785] LustreError: 156976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784590110 with bad export cookie 11166410052983978175 [ 6473.632332] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6474.127557] Lustre: server umount lustre-MDT0001 complete [ 6477.592234] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 19:28:32 (1784590112) [ 6479.317470] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6481.117994] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 19:28:36 (1784590116) [ 6488.393384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6488.832985] LustreError: 161361:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6488.847428] LustreError: 161361:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6488.893815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6493.405457] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 6494.194670] LustreError: 161362:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6498.282130] LustreError: 161361:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6503.261861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6503.399050] LustreError: 161362:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6510.193521] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 6519.373202] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6526.266335] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6533.419186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6533.854385] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6533.857402] Lustre: Skipped 1 previous similar message [ 6540.473251] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 6549.084420] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6559.714221] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6559.722418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6559.729908] Lustre: Skipped 2 previous similar messages [ 6559.750216] Lustre: Skipped 4 previous similar messages [ 6561.080758] Lustre: server umount lustre-MDT0000 complete [ 6564.837042] LustreError: 161362:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6564.881632] LustreError: 161362:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6566.140701] LustreError: 161342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784590202 with bad export cookie 11166410052983979057 [ 6566.146090] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6566.157469] LustreError: 161342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6569.954252] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6569.992355] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6569.997851] Lustre: Skipped 1 previous similar message [ 6572.692579] Lustre: server umount lustre-MDT0001 complete [ 6581.127829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6593.474981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6593.819909] LustreError: 165204:0:(llog.c:1655:llog_backup()) MGC192.168.201.159@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6593.834234] Lustre: 165204:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.159@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6596.578443] LustreError: 165208:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6596.593726] LustreError: 165208:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6616.873953] LustreError: 165211:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6616.901965] LustreError: 165211:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [ 6651.683132] LustreError: 165208:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6651.698968] LustreError: 165208:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 25 previous similar messages [ 6654.432280] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6654.437345] Lustre: 165204:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.159@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6695.428249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6700.529095] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6700.589192] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 6711.417289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6711.781294] LustreError: 148417:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000079c33c9d x1871278025867392/t0(0) o250->MGC192.168.201.159@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6716.452884] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 6717.430255] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6724.264866] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6726.362355] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6733.758272] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6735.753235] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6758.369687] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6758.377296] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6758.399663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6758.423475] Lustre: Skipped 2 previous similar messages [ 6758.449546] Lustre: Skipped 4 previous similar messages [ 6765.164525] Lustre: server umount lustre-MDT0000 complete [ 6768.610070] LustreError: 165227:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6768.632649] LustreError: 165227:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 46 previous similar messages [ 6768.825236] LustreError: 161342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784590405 with bad export cookie 11166410052983985511 [ 6768.828587] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6768.843479] LustreError: 161342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6769.371397] Lustre: server umount lustre-MDT0001 complete [ 6784.100977] Lustre: server umount lustre-OST0000 complete [ 6807.794741] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [ 6819.551644] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 6861.445242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6861.931460] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6862.013507] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6862.128465] Lustre: lustre-MDT0000: new disk, initializing [ 6862.276979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6862.284685] Lustre: Skipped 1 previous similar message [ 6862.335239] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6869.243262] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 6881.968050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6882.110205] Lustre: 170312:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6882.664725] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6888.574292] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 6900.215466] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6906.743909] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6914.843868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6915.084335] Lustre: lustre-OST0000: new disk, initializing [ 6915.086993] Lustre: Skipped 1 previous similar message [ 6915.090446] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6915.093726] Lustre: Skipped 2 previous similar messages [ 6915.097886] Lustre: 172282:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6922.736904] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 6924.878651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6924.886116] Lustre: Skipped 1 previous similar message [ 6924.889339] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6924.998883] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6932.974993] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6938.435097] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6938.769587] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6944.207573] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6944.471928] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6950.882113] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6950.908438] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6950.911230] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6950.920742] Lustre: Skipped 1 previous similar message [ 6950.949664] Lustre: Skipped 4 previous similar messages [ 6952.789808] Lustre: server umount lustre-OST0000 complete [ 6959.585264] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6959.611147] LustreError: Skipped 1 previous similar message [ 6963.586602] Lustre: server umount lustre-MDT0000 complete [ 6966.246599] LustreError: 170324:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6966.261872] LustreError: 170324:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6967.840795] LustreError: 170306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784590604 with bad export cookie 11166410052984002346 [ 6967.849291] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6967.861263] LustreError: 170306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6968.213980] Lustre: server umount lustre-MDT0001 complete [ 6979.661847] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 19:36:54 (1784590614) [ 6987.101109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6993.029782] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 7003.281976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7009.071725] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 7017.751084] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7023.809932] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7032.508745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7040.640514] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 7050.904824] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7073.463204] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 19:38:28 (1784590708) [ 7080.419020] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7080.435802] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7080.455885] Lustre: Skipped 2 previous similar messages [ 7080.460258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7080.462738] Lustre: Skipped 3 previous similar messages [ 7085.084964] Lustre: server umount lustre-OST0000 complete [ 7096.741699] Lustre: server umount lustre-MDT0000 complete [ 7100.778877] LustreError: 175431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784590737 with bad export cookie 11166410052984003228 [ 7100.784809] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7100.793072] LustreError: 175431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7101.190367] Lustre: server umount lustre-MDT0001 complete [ 7111.998307] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 7115.073471] Key type lgssc unregistered [ 7115.450645] LNet: 178449:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7115.454936] LNetError: 178449:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7115.483818] LNet: Removed LNI 192.168.201.159@tcp [ 7116.488824] Key type .llcrypt unregistered [ 7116.490673] Key type ._llcrypt unregistered [ 7139.214288] Key type ._llcrypt registered [ 7139.218295] Key type .llcrypt registered [ 7139.435403] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 7140.802021] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7141.051426] alg: No test for adler32 (adler32-zlib) [ 7142.262639] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 7142.634185] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 7144.312569] Key type lgssc registered [ 7145.811913] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7154.350701] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7154.364748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7170.017441] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7175.137540] LustreError: 179694:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.159@tcp: failed processing log, type 4: rc = -110 [ 7180.330242] LustreError: 179694:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7180.347286] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7180.371680] LustreError: MGC192.168.201.159@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7180.384051] LustreError: 179694:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7180.391527] LustreError: 179694:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7180.397129] LustreError: 179694:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7180.468460] Lustre: server umount lustre-OST0000 complete [ 7180.474486] LustreError: 179694:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7189.884924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7191.624348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7197.615483] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 7201.775793] LustreError: 180193:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7201.804649] LustreError: 180193:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7206.890095] LustreError: 180192:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7208.085865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7208.672880] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7214.298832] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 7222.218432] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7228.030240] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7235.526827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7235.709173] LustreError: 182009:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7235.717985] Lustre: 182009:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.201.159@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7235.859679] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7242.749226] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 7250.699542] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7254.377355] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7254.381824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7254.398489] Lustre: Skipped 2 previous similar messages [ 7254.420198] Lustre: Skipped 2 previous similar messages [ 7259.622296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7259.626545] Lustre: Skipped 2 previous similar messages [ 7259.889909] Lustre: server umount lustre-MDT0000 complete [ 7263.820127] LustreError: 180173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784590900 with bad export cookie 8035326323558182350 [ 7263.823361] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7263.830326] LustreError: 180173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7264.672374] LustreError: 180921:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000f4513515 x1871279060711296/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7264.744678] LustreError: 180188:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7264.746925] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7264.911235] Lustre: server umount lustre-MDT0001 complete [ 7271.732297] Lustre: server umount lustre-OST0000 complete [ 7281.976660] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [ 7291.965261] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 7333.391291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7333.676784] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7333.739360] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7333.826743] Lustre: lustre-MDT0000: new disk, initializing [ 7333.981115] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7334.017703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7339.256676] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 7351.201583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7351.375755] Lustre: 185533:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7351.433689] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7351.437337] Lustre: Skipped 1 previous similar message [ 7351.528193] Lustre: lustre-MDT0001: new disk, initializing [ 7351.648680] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7351.704838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7351.722738] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7356.997778] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 7369.531035] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7376.397872] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7385.115339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7385.505423] Lustre: lustre-OST0000: new disk, initializing [ 7385.508671] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7385.516420] Lustre: 187502:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7385.602608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7391.813300] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7391.836786] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7391.998134] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7393.600615] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 7404.707480] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7410.238979] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7410.656259] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7416.736409] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7416.982388] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7422.958399] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7422.968554] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7422.997964] LustreError: Skipped 1 previous similar message [ 7423.022842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7423.026963] Lustre: Skipped 1 previous similar message [ 7425.442291] Lustre: server umount lustre-OST0000 complete [ 7433.190468] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7433.201027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7433.202370] Lustre: Skipped 2 previous similar messages [ 7433.208928] Lustre: Skipped 1 previous similar message [ 7436.244452] Lustre: server umount lustre-MDT0000 complete [ 7438.315506] LustreError: 186695:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7438.342214] LustreError: 186695:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7440.709222] LustreError: 185523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784591077 with bad export cookie 8035326323558183071 [ 7440.720423] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7440.740507] LustreError: 185523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7441.434798] Lustre: server umount lustre-MDT0001 complete [ 7453.235714] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 19:44:47 (1784591087) [ 7476.073886] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 7479.565249] Key type lgssc unregistered [ 7479.836813] LNet: 190450:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7479.853900] LNetError: 190450:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7480.879101] LNet: Removed LNI 192.168.201.159@tcp [ 7482.217589] Key type .llcrypt unregistered [ 7482.219654] Key type ._llcrypt unregistered [ 7505.391620] Key type ._llcrypt registered [ 7505.401140] Key type .llcrypt registered [ 7505.664466] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 7507.330144] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7507.401077] alg: No test for adler32 (adler32-zlib) [ 7508.571377] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 7508.856646] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 7510.573319] Key type lgssc registered [ 7511.992149] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7520.151504] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7520.178366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7521.638726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7526.606512] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 7532.009798] LustreError: 191670:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7532.030349] LustreError: 191670:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7535.568902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7536.198839] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7541.118668] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 7549.725426] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7555.878302] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7564.604523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7564.863661] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7564.870547] LustreError: 193535:0:(llog.c:1696:llog_backup()) MGC192.168.201.159@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7564.882256] Lustre: 193535:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.201.159@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7564.996980] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7572.161526] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 7580.780397] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7585.767168] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7585.772412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7585.797815] Lustre: Skipped 2 previous similar messages [ 7585.824945] Lustre: Skipped 2 previous similar messages [ 7589.384325] Lustre: server umount lustre-MDT0000 complete [ 7590.883861] LustreError: 191664:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7590.913605] LustreError: 191664:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7592.782969] LustreError: 191650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784591229 with bad export cookie 12514776402790298534 [ 7592.798555] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7593.103653] Lustre: server umount lustre-MDT0001 complete [ 7606.486388] Lustre: server umount lustre-OST0000 complete [ 7615.778598] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [ 7628.320821] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 7667.699498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7668.074321] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7668.144283] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7668.250639] Lustre: lustre-MDT0000: new disk, initializing [ 7668.345350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7668.363482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7673.360720] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 7684.767393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7684.885698] Lustre: 197057:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7684.919922] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7684.922915] Lustre: Skipped 1 previous similar message [ 7684.926512] LustreError: 197842:0:(llog.c:1696:llog_backup()) MGC192.168.201.159@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7684.935350] LustreError: 197842:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7684.940364] Lustre: 197842:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.159@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7684.971346] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7684.978939] Lustre: Skipped 1 previous similar message [ 7685.122466] Lustre: lustre-MDT0001: new disk, initializing [ 7685.230524] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7685.272843] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7685.283467] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7690.114883] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 7700.326486] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7706.616230] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7714.063544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7714.287978] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7714.293870] Lustre: Skipped 1 previous similar message [ 7714.298900] LustreError: 199004:0:(llog.c:1696:llog_backup()) MGC192.168.201.159@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7714.305619] LustreError: 199004:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7714.310356] Lustre: 199004:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.159@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7714.318994] Lustre: 199004:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7714.334616] Lustre: lustre-OST0000: new disk, initializing [ 7714.337810] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7714.352773] Lustre: 199029:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7714.449686] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7720.530360] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 7722.600174] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7722.612191] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7722.737933] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7730.570343] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7736.439680] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7736.698418] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7741.012935] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7741.217196] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7743.457669] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7743.462906] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7743.478790] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7743.485085] Lustre: Skipped 1 previous similar message [ 7748.582027] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7748.588449] Lustre: Skipped 2 previous similar messages [ 7753.704626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7753.722825] Lustre: Skipped 1 previous similar message [ 7757.280287] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7757.560559] Lustre: server umount lustre-OST0000 complete [ 7761.893125] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7761.902440] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7761.919148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7763.952185] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7767.991232] Lustre: server umount lustre-MDT0000 complete [ 7769.065150] LustreError: 197065:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7769.079372] LustreError: 197065:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7771.870946] LustreError: 197049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784591408 with bad export cookie 12514776402790299255 [ 7771.873527] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7771.886084] LustreError: 197049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7772.211142] Lustre: server umount lustre-MDT0001 complete [ 7781.424766] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 19:50:16 (1784591416) [ 7799.081603] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 7809.832920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7810.193525] LustreError: 202162:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7810.246950] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7814.372441] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 7815.652534] LustreError: 202163:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7823.325297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7828.371829] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 7831.343462] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7838.166806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7844.478702] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 7854.116895] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7854.319803] Lustre: lustre-OST0001: new disk, initializing [ 7854.325694] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7854.331706] Lustre: 204468:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7854.439868] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7854.445218] Lustre: Skipped 2 previous similar messages [ 7861.896760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7861.905776] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7861.976431] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 7862.030984] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7873.906563] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7880.757586] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7882.632680] Lustre: DEBUG MARKER: ADD OST3 [ 7888.103709] loop: module loaded [ 7888.226265] loop0: detected capacity change from 0 to 204800000 [ 7888.458581] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7888.478442] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7888.503421] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7895.150366] loop0: detected capacity change from 0 to 204800000 [ 7895.598530] Lustre: DEBUG MARKER: STOP OST3 [ 7902.039207] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7902.302646] Lustre: lustre-OST0003: new disk, initializing [ 7902.306801] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7902.314037] Lustre: 206508:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7903.797311] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7903.929934] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7903.937312] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7903.991305] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7905.528840] Lustre: Failing over lustre-OST0003 [ 7905.824213] LustreError: 206638:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7905.830366] Lustre: server umount lustre-OST0003 complete [ 7909.089675] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 7909.190080] LustreError: 204510:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.201.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7909.203789] LustreError: 204510:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7909.351075] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7909.358881] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7918.509742] Lustre: DEBUG MARKER: START OST3 again [ 7926.612080] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7926.998321] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7927.009730] Lustre: Skipped 1 previous similar message [ 7927.029924] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7928.713641] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7928.762379] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7928.766437] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7934.021721] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 7955.171437] Lustre: Failing over lustre-OST0003 [ 7955.280848] Lustre: server umount lustre-OST0003 complete [ 7958.003324] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7958.018223] Lustre: Skipped 1 previous similar message [ 7958.024086] LustreError: 203647:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7958.045760] LustreError: 203647:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 7973.383924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7973.391690] Lustre: Skipped 3 previous similar messages [ 7979.792771] Lustre: server umount lustre-MDT0000 complete [ 7984.298833] LustreError: 202143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784591621 with bad export cookie 12514776402790300137 [ 7984.313055] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7988.719149] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7988.760808] Lustre: Skipped 6 previous similar messages [ 7989.730070] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7989.740476] Lustre: Skipped 9 previous similar messages [ 7990.806256] Lustre: server umount lustre-MDT0001 complete [ 7995.802433] Lustre: server umount lustre-OST0000 complete [ 8000.056878] Lustre: server umount lustre-OST0001 complete [ 8006.193967] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [ 8013.669862] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 8051.706679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8051.904399] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8051.924798] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8051.987812] Lustre: lustre-MDT0000: new disk, initializing [ 8052.052632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8056.671653] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8066.604931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8066.695764] Lustre: 211933:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8066.891688] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8066.896131] Lustre: Skipped 1 previous similar message [ 8066.957310] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8071.119716] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8079.762775] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8085.311701] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8091.751124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8092.082877] Lustre: lustre-OST0000: new disk, initializing [ 8092.087832] Lustre: Skipped 1 previous similar message [ 8092.094905] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8092.103655] Lustre: Skipped 2 previous similar messages [ 8092.112797] Lustre: 213902:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8094.132313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8094.143150] Lustre: Skipped 1 previous similar message [ 8094.155051] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8094.326178] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8099.320602] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8109.223199] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8113.843891] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8114.061460] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8119.864857] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8120.149985] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8124.902838] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8124.928719] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8124.932523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8124.937523] LustreError: Skipped 1 previous similar message [ 8124.956203] Lustre: Skipped 1 previous similar message [ 8128.982929] Lustre: server umount lustre-OST0000 complete [ 8133.602132] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8138.939735] Lustre: server umount lustre-MDT0000 complete [ 8140.271498] LustreError: 212793:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8140.295463] LustreError: 212793:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 16 previous similar messages [ 8142.875077] LustreError: 211928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784591779 with bad export cookie 12514776402790302433 [ 8142.892755] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8142.896545] LustreError: 211928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 8143.370603] Lustre: server umount lustre-MDT0001 complete [ 8147.184997] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 19:56:22 (1784591782) [ 8164.813547] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [ 8174.034943] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 8209.209815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8209.515109] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8209.542895] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8209.613288] Lustre: lustre-MDT0000: new disk, initializing [ 8209.710176] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8214.399255] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8224.748922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8224.880821] Lustre: 218509:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8225.066088] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8229.930885] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8239.450325] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8246.070783] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8252.907356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8253.163972] Lustre: 220480:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8255.053838] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8260.387248] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8269.097338] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8273.461608] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8273.637577] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8277.908541] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8278.069729] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8280.545278] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8280.552322] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8280.556444] LustreError: Skipped 1 previous similar message [ 8280.557472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8280.557477] Lustre: Skipped 3 previous similar messages [ 8280.586564] Lustre: Skipped 4 previous similar messages [ 8286.201343] Lustre: server umount lustre-OST0000 complete [ 8296.426810] Lustre: server umount lustre-MDT0000 complete [ 8299.434083] LustreError: 218502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784591936 with bad export cookie 12514776402790303315 [ 8299.441634] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8299.445325] LustreError: 218502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8299.711430] Lustre: server umount lustre-MDT0001 complete [ 8307.165256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8307.456368] LustreError: 222696:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8311.802810] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8319.193258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8323.586451] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8329.921724] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8334.636773] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8341.517976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8342.030338] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8342.034394] Lustre: Skipped 6 previous similar messages [ 8348.075717] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8355.267690] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8365.539936] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8370.671853] Lustre: server umount lustre-OST0000 complete [ 8380.625240] Lustre: server umount lustre-MDT0000 complete [ 8383.619834] LustreError: 222678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784592020 with bad export cookie 12514776402790304197 [ 8383.635781] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8383.895247] Lustre: server umount lustre-MDT0001 complete [ 8495.010564] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 8497.275126] Key type lgssc unregistered [ 8497.537685] LNet: 226371:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8497.542582] LNetError: 226371:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8497.558924] LNet: Removed LNI 192.168.201.159@tcp [ 8498.476191] Key type .llcrypt unregistered [ 8498.478913] Key type ._llcrypt unregistered [ 8521.533469] Key type ._llcrypt registered [ 8521.536455] Key type .llcrypt registered [ 8521.622552] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [ 8532.735117] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 8533.818798] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8533.832777] alg: No test for adler32 (adler32-zlib) [ 8534.922121] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 8535.092222] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 8536.725658] Key type lgssc registered [ 8537.876310] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8567.203568] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8567.214971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8568.380357] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8568.399459] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8568.460570] Lustre: lustre-MDT0000: new disk, initializing [ 8568.506529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8568.519665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8571.458290] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8579.430233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8579.531516] Lustre: 229704:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8579.573837] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8579.576867] Lustre: Skipped 1 previous similar message [ 8579.624289] Lustre: lustre-MDT0001: new disk, initializing [ 8579.689248] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8579.734426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8579.743763] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8582.821757] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8589.747766] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8594.401635] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8599.335287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8599.516450] Lustre: lustre-OST0000: new disk, initializing [ 8599.523493] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8599.531774] Lustre: 231669:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8599.612959] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8604.787780] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8607.284564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8607.289463] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8607.431991] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8611.726193] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8615.051764] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8615.173550] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8618.281171] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8618.432492] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8623.079879] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8623.088443] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8623.091766] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8623.092641] LustreError: Skipped 1 previous similar message [ 8625.687535] Lustre: server umount lustre-OST0000 complete [ 8630.753450] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8630.761334] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8630.776284] Lustre: Skipped 1 previous similar message [ 8630.787890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8630.791904] Lustre: Skipped 1 previous similar message [ 8633.313310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8633.315837] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8634.783638] Lustre: server umount lustre-MDT0000 complete [ 8637.500390] LustreError: 229696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784592274 with bad export cookie 2135556671461855045 [ 8637.508799] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8637.510648] LustreError: 229696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8637.704685] Lustre: server umount lustre-MDT0001 complete [ 8650.370393] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 8658.012213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8658.361523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8661.682285] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8663.521427] LustreError: 234600:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8663.531390] LustreError: 234600:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8667.594845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8667.807343] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8671.256739] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8673.253845] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8678.706054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8678.944507] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8683.489108] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8689.852188] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8689.982526] Lustre: lustre-OST0001: new disk, initializing [ 8689.985129] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8689.989105] Lustre: 236903:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8693.890784] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8699.504509] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8699.519547] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8699.617167] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8703.238834] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8707.415575] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8716.404314] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 20:05:52 (1784592352) [ 8719.329439] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8719.334615] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8719.355184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8719.359331] Lustre: Skipped 1 previous similar message [ 8724.179757] Lustre: server umount lustre-MDT0000 complete [ 8725.477768] LustreError: 234596:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8725.489711] LustreError: 234596:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8726.625929] LustreError: 234579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784592363 with bad export cookie 2135556671461855927 [ 8726.628988] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8726.631152] LustreError: 234579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8726.905133] Lustre: server umount lustre-MDT0001 complete [ 8739.972151] Lustre: server umount lustre-OST0000 complete [ 8751.449773] Lustre: server umount lustre-OST0001 complete [ 8756.427500] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [ 8762.911950] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 8785.742991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8785.943981] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8785.963263] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8786.011311] Lustre: lustre-MDT0000: new disk, initializing [ 8786.061914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8786.065534] Lustre: Skipped 1 previous similar message [ 8786.075955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8788.656113] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8796.122475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8796.202901] Lustre: 240847:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8796.242786] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8796.246152] Lustre: Skipped 1 previous similar message [ 8796.306742] Lustre: lustre-MDT0001: new disk, initializing [ 8796.395627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8796.399550] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8798.764667] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8805.033126] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8808.652764] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8813.173926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8813.362520] Lustre: lustre-OST0000: new disk, initializing [ 8813.366372] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8813.371911] Lustre: 242813:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8815.445740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8815.450198] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8815.559782] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8817.696852] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8824.053758] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8826.955353] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8827.109347] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8830.202248] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8830.387861] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8836.068902] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8836.072820] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8836.074941] LustreError: Skipped 1 previous similar message [ 8836.076096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8836.084793] Lustre: Skipped 4 previous similar messages [ 8836.090474] Lustre: Skipped 5 previous similar messages [ 8846.304208] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8846.309611] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8846.316437] Lustre: Skipped 3 previous similar messages [ 8846.420630] Lustre: server umount lustre-OST0000 complete [ 8851.429374] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8863.204352] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8863.428732] Lustre: server umount lustre-MDT0000 complete [ 8865.855405] LustreError: 240840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784592502 with bad export cookie 2135556671461858048 [ 8865.861087] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8865.866622] LustreError: 240840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8866.110231] Lustre: server umount lustre-MDT0001 complete [ 8871.806290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8872.135485] LustreError: 245029:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8872.216229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8872.231774] Lustre: Skipped 2 previous similar messages [ 8875.173620] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8877.543326] LustreError: 245030:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8881.203206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8884.656644] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8889.742025] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8893.471327] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8897.610022] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8901.341724] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 8906.528734] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9051.618431] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9051.622980] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9051.634101] Lustre: Skipped 1 previous similar message [ 9051.637395] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9051.641141] Lustre: Skipped 6 previous similar messages [ 9057.244901] Lustre: server umount lustre-OST0000 complete [ 9065.638814] Lustre: server umount lustre-MDT0000 complete [ 9068.161358] LustreError: 245010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784592705 with bad export cookie 2135556671461858930 [ 9068.165294] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9068.168280] LustreError: 245010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9068.365792] Lustre: server umount lustre-MDT0001 complete [ 9172.913534] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [ 9174.525667] Key type lgssc unregistered [ 9174.706518] LNet: 248633:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9174.710571] LNetError: 248633:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9174.724365] LNet: Removed LNI 192.168.201.159@tcp [ 9175.295455] Key type .llcrypt unregistered [ 9175.303595] Key type ._llcrypt unregistered [ 9191.193316] Key type ._llcrypt registered [ 9191.196098] Key type .llcrypt registered [ 9191.272690] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [ 9200.485667] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 9200.940399] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9200.989958] alg: No test for adler32 (adler32-zlib) [ 9201.936587] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 9202.063249] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [ 9203.680473] Key type lgssc registered [ 9204.353642] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9231.732806] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 9237.943244] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9237.956410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9239.088081] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9239.116080] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9239.182565] Lustre: lustre-MDT0000: new disk, initializing [ 9239.236435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9239.250326] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9241.377616] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9247.877292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9247.936702] Lustre: 252676:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9247.971871] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9247.974313] Lustre: Skipped 1 previous similar message [ 9248.016669] Lustre: lustre-MDT0001: new disk, initializing [ 9248.059600] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9248.080718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9248.087195] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9250.570453] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9254.142816] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9258.508539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9258.657760] Lustre: lustre-OST0000: new disk, initializing [ 9258.659701] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9258.662993] Lustre: 254307:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9258.706165] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9262.116550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9262.123990] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9262.211117] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9262.416181] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9269.405935] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9269.522459] Lustre: lustre-OST0001: new disk, initializing [ 9269.526436] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9269.536781] Lustre: 255179:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9269.608371] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9272.659479] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9277.464579] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9277.477472] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9277.527251] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9279.612427] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9285.342267] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9292.775044] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9292.782716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9292.791831] Lustre: Skipped 2 previous similar messages [ 9292.802955] Lustre: Skipped 3 previous similar messages [ 9293.551944] Lustre: server umount lustre-MDT0000 complete [ 9295.925497] LustreError: 252668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784592932 with bad export cookie 18377256393463783674 [ 9295.928797] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9295.937811] LustreError: 252668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9296.250996] Lustre: server umount lustre-MDT0001 complete [ 9309.142814] Lustre: server umount lustre-OST0000 complete [ 9320.460397] Lustre: server umount lustre-OST0001 complete [ 9328.528985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9333.926468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9339.074314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9344.879198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9355.991559] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 9361.388819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9361.397751] Lustre: lustre-MDT0000: reset Object Index mappings [ 9361.676069] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9363.922626] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9367.012321] LustreError: 260183:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9367.024875] LustreError: 260183:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9368.598957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9368.609520] Lustre: lustre-MDT0001: reset Object Index mappings [ 9371.221670] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9372.817137] Lustre: 261327:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9376.879214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9377.105780] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9377.111040] Lustre: Skipped 1 previous similar message [ 9379.178572] LustreError: 261682:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9380.160780] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9384.424272] LustreError: 261683:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9384.974547] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9387.180039] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9387.195379] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9388.951268] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9393.577808] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9395.594902] Lustre: 263196:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9397.736481] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9397.742030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9397.748303] Lustre: Skipped 3 previous similar messages [ 9397.751053] Lustre: Skipped 1 previous similar message [ 9402.856371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9402.863244] Lustre: Skipped 5 previous similar messages [ 9404.008509] Lustre: server umount lustre-MDT0000 complete [ 9406.441804] LustreError: 260163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784593043 with bad export cookie 18377256393463790464 [ 9406.450450] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9406.452638] LustreError: 260163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9406.726539] Lustre: server umount lustre-MDT0001 complete [ 9420.194834] Lustre: server umount lustre-OST0000 complete [ 9431.383612] Lustre: server umount lustre-OST0001 complete [ 9442.733081] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 20:17:58 (1784593078) [ 9452.972627] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [ 9458.637320] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 9477.219982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9477.338084] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9477.358175] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9477.409834] Lustre: lustre-MDT0000: new disk, initializing [ 9477.472920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9477.477217] Lustre: Skipped 1 previous similar message [ 9477.497602] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9479.814950] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9487.233158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9487.330956] Lustre: 267445:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9487.369809] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9487.372516] Lustre: Skipped 1 previous similar message [ 9487.419638] Lustre: lustre-MDT0001: new disk, initializing [ 9487.503246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9487.512464] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9490.323676] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9496.197988] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9498.923839] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9502.255907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9502.447143] Lustre: 269411:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9503.699846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9503.705140] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9503.759582] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9505.608445] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9510.727733] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9513.055140] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9513.204588] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9515.675343] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9515.818876] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9519.073272] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9519.076974] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9519.081445] LustreError: Skipped 1 previous similar message [ 9519.099373] Lustre: Skipped 1 previous similar message [ 9519.100524] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9524.201182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9524.208301] Lustre: Skipped 2 previous similar messages [ 9531.361145] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9531.474321] Lustre: server umount lustre-OST0000 complete [ 9533.923026] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9533.926365] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9533.932220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9533.934896] Lustre: Skipped 2 previous similar messages [ 9539.873465] Lustre: server umount lustre-MDT0000 complete [ 9542.200048] LustreError: 267437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784593179 with bad export cookie 18377256393463792550 [ 9542.205487] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9542.207733] LustreError: 267437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9542.400740] Lustre: server umount lustre-MDT0001 complete [ 9553.089325] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 9559.085335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9559.275274] LustreError: 272344:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9559.320344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9559.323618] Lustre: Skipped 2 previous similar messages [ 9561.842565] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9564.643453] LustreError: 272345:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9566.727897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9569.276236] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9570.965886] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9574.413919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9577.604508] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9581.286546] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9581.370501] Lustre: lustre-OST0001: new disk, initializing [ 9581.372832] Lustre: Skipped 1 previous similar message [ 9581.376728] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9581.379809] Lustre: Skipped 1 previous similar message [ 9581.382815] Lustre: 274648:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9582.674983] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9582.679853] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9582.734406] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9583.772293] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9589.381253] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9591.008463] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9593.425354] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 9593.445896] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 9597.921174] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9597.924566] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9597.930270] Lustre: Skipped 1 previous similar message [ 9597.932808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9597.935363] Lustre: Skipped 4 previous similar messages [ 9602.035564] Lustre: server umount lustre-MDT0000 complete [ 9604.000546] LustreError: 272326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784593240 with bad export cookie 18377256393463793432 [ 9604.001929] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9604.009245] LustreError: 272326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9604.179267] Lustre: server umount lustre-MDT0001 complete [ 9616.407280] Lustre: server umount lustre-OST0000 complete [ 9629.124750] Lustre: server umount lustre-OST0001 complete [ 9638.545215] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 9644.928402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9645.161319] LustreError: 277701:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9645.200570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9645.204385] Lustre: Skipped 3 previous similar messages [ 9647.459643] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9652.718852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9655.817719] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9657.665640] Lustre: 278842:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9661.429167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9662.629340] LustreError: 279193:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9662.637484] LustreError: 279193:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9664.665031] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9669.268133] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9670.457406] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 9670.472575] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 9672.826527] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9677.680431] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9679.634746] Lustre: 280719:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9682.444532] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 9682.457207] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 9688.199346] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 20:22:03 (1784593323) [ 9701.347399] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9701.348360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9701.363342] Lustre: Skipped 6 previous similar messages [ 9701.372261] Lustre: Skipped 7 previous similar messages [ 9703.351702] Lustre: server umount lustre-MDT0000 complete [ 9705.704377] LustreError: 278444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784593342 with bad export cookie 18377256393463799746 [ 9705.708544] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9705.715324] LustreError: 278444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9706.470322] LustreError: 280023:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9706.482816] LustreError: 280023:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9708.028736] Lustre: server umount lustre-MDT0001 complete [ 9716.566227] Lustre: server umount lustre-OST0000 complete [ 9733.088232] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9733.218515] Lustre: server umount lustre-OST0001 complete [ 9737.711761] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [ 9743.468348] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 9772.350326] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [ 9778.257839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9778.401520] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9778.418683] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9778.464887] Lustre: lustre-MDT0000: new disk, initializing [ 9778.542142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9778.550483] Lustre: Skipped 3 previous similar messages [ 9778.577153] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9781.089911] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9787.580365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9787.687122] Lustre: 284954:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9787.891488] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9790.702861] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9798.180785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9798.441915] Lustre: 286584:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9799.671133] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9802.251419] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9809.191239] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9809.338170] Lustre: 287458:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9810.862568] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9812.893539] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9819.268293] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9821.566183] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9821.569679] Lustre: Skipped 1 previous similar message [ 9829.768397] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 20:24:25 (1784593465) [ 9831.399126] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9831.402550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9831.406697] Lustre: Skipped 5 previous similar messages [ 9831.414084] Lustre: Skipped 5 previous similar messages [ 9837.345996] Lustre: server umount lustre-MDT0000 complete [ 9839.990968] LustreError: 284947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784593476 with bad export cookie 18377256393463807621 [ 9839.991645] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9839.999757] LustreError: 284947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9840.246486] Lustre: server umount lustre-MDT0001 complete [ 9852.329563] Lustre: server umount lustre-OST0000 complete [ 9865.087536] Lustre: server umount lustre-OST0001 complete [ 9870.083968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9870.301423] LustreError: 289701:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9872.814661] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9877.771217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9880.289409] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9884.283664] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9887.500237] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9891.243514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9895.079462] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9899.174609] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9912.718320] loop1: detected capacity change from 0 to 204800000 [ 9912.868955] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9912.879086] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9912.884767] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9915.505343] loop1: detected capacity change from 0 to 204800000 [ 9918.626191] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9918.681235] LustreError: 289688:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs13363: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 9918.686875] LustreError: 289688:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs13363 [ 9918.689160] LustreError: 289688:0:(mgs_handler.c:517:mgs_target_reg()) fs13363-MDT0001: Error checking target index: rc = -13 [ 9918.693683] LustreError: fs13363-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 9918.697666] LustreError: 293413:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 9918.701139] LustreError: 293413:0:(tgt_mount.c:2129:server_put_super()) no obd fs13363-MDT0001 [ 9918.704097] LustreError: 293413:0:(tgt_mount.c:116:server_deregister_mount()) fs13363-MDT0001 not registered [ 9918.725804] Lustre: server umount fs13363-MDT0001 complete [ 9918.727491] LustreError: 293413:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 9923.025500] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9923.074635] Lustre: Setting parameter fs13363-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs13363-MDT0001 [ 9923.105431] Lustre: srv-fs13363-MDT0001: No data found on store. Initialize space. [ 9923.112680] Lustre: Skipped 4 previous similar messages [ 9923.164541] Lustre: fs13363-MDT0001: new disk, initializing [ 9923.166254] Lustre: Skipped 3 previous similar messages [ 9923.230662] Lustre: srv-fs13363-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 9923.235160] Lustre: 293879:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs13363-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9923.244176] LustreError: 293879:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs13363-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9925.634626] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9929.904978] Lustre: server umount fs13363-MDT0001 complete [ 9937.597994] loop1: detected capacity change from 0 to 204800000 [ 9940.814745] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9940.991501] Lustre: 295444:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs13363-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9940.994473] LustreError: 295444:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs13363-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9943.744913] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9946.562448] Lustre: server umount fs13363-MDT0001 complete [ 9954.279047] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9954.288336] Lustre: Skipped 2 previous similar messages [ 9958.428304] Lustre: server umount lustre-MDT0000 complete [ 9963.790086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9963.869444] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9966.462833] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9969.130241] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9971.023497] loop1: detected capacity change from 0 to 204800000 [ 9974.505033] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9974.719797] Lustre: 297888:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs13363-MDT0001-mdtlov: can't create new llog: rc = -115 [ 9974.723716] LustreError: 297888:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs13363-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 9977.446539] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [ 9980.473462] Lustre: server umount fs13363-MDT0001 complete [ 9984.485529] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [ 9994.721207] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9994.733519] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9994.737112] Lustre: Skipped 9 previous similar messages [ 9999.848705] LustreError: 291531:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9999.855242] LustreError: 291531:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 63 previous similar messages [10007.008166] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10007.101479] Lustre: server umount lustre-OST0000 complete [10010.087298] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10015.552514] Lustre: server umount lustre-MDT0000 complete [10017.610310] LustreError: 289683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784593654 with bad export cookie 18377256393463810309 [10017.615098] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10017.619885] LustreError: 289683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10017.810142] Lustre: server umount lustre-MDT0001 complete [10023.641633] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [10025.179350] Key type lgssc unregistered [10025.439709] LNet: 300236:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10025.448967] LNetError: 300236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10026.470527] LNet: Removed LNI 192.168.201.159@tcp [10027.071183] Key type .llcrypt unregistered [10027.072738] Key type ._llcrypt unregistered [10039.315680] Key type ._llcrypt registered [10039.317962] Key type .llcrypt registered [10039.438024] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [10040.163845] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10040.183169] alg: No test for adler32 (adler32-zlib) [10041.120203] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [10041.302889] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [10042.928572] Key type lgssc registered [10043.554448] Lustre: Echo OBD driver; http://www.lustre.org/ [10048.017674] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10048.034637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10049.298876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10051.654444] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [10056.584165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10056.837993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10059.868678] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [10064.821985] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10068.551112] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10072.960405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10073.220659] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10074.281593] LustreError: 303313:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10074.294589] LustreError: 303313:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10076.788847] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [10079.720693] LustreError: 303312:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10081.272233] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10081.766244] LustreError: 303312:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10081.778411] LustreError: 303312:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10086.889261] LustreError: 303416:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10086.908327] LustreError: 303416:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10092.006348] LustreError: 303416:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10092.022842] LustreError: 303416:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10094.002105] loop1: detected capacity change from 0 to 204800000 [10094.148420] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10094.158192] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10094.168342] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10097.519361] loop1: detected capacity change from 0 to 204800000 [10101.093064] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10101.190168] Lustre: fs22583-OST0000: new disk, initializing [10101.193195] Lustre: srv-fs22583-OST0000: No data found on store. Initialize space. [10101.195770] Lustre: 305216:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10101.251148] Lustre: fs22583-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10102.243399] LustreError: 303313:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10102.258543] LustreError: 303313:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [10105.201402] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [10111.053830] Lustre: server umount fs22583-OST0000 complete [10118.420860] loop1: detected capacity change from 0 to 204800000 [10119.463052] LustreError: 303313:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10119.473591] LustreError: 303313:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [10121.766458] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10121.864312] Lustre: fs22583-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10124.818579] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [10129.204790] loop2: detected capacity change from 0 to 204800000 [10129.348122] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10129.355052] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10129.360108] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10132.069054] loop2: detected capacity change from 0 to 204800000 [10134.954842] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10135.000171] LustreError: fs22583-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10135.006690] LustreError: fs22583-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10135.011430] LustreError: 308253:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10135.015851] LustreError: 308253:0:(tgt_mount.c:2129:server_put_super()) no obd fs22583-OST0001 [10135.019697] LustreError: 308253:0:(tgt_mount.c:116:server_deregister_mount()) fs22583-OST0001 not registered [10135.047958] Lustre: server umount fs22583-OST0001 complete [10135.050750] LustreError: 308253:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [10139.862648] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10139.940273] Lustre: fs22583-OST0001: new disk, initializing [10139.942804] Lustre: srv-fs22583-OST0001: No data found on store. Initialize space. [10139.946792] Lustre: 308804:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10139.977976] Lustre: fs22583-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10143.325785] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [10148.612943] Lustre: server umount fs22583-OST0000 complete [10153.448139] LustreError: 303314:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10153.464077] LustreError: 303314:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 19 previous similar messages [10154.081130] Lustre: server umount fs22583-OST0001 complete [10158.567117] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10160.719825] loop1: detected capacity change from 0 to 204800000 [10163.839335] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10163.976286] Lustre: fs22583-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10167.217539] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [10168.808083] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10170.560076] loop2: detected capacity change from 0 to 204800000 [10172.939085] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10175.707321] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [10179.467718] Lustre: server umount fs22583-OST0000 complete [10183.249835] Lustre: server umount fs22583-OST0001 complete [10189.282288] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10189.285547] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10189.292606] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10189.295022] Lustre: Skipped 1 previous similar message [10194.401466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10194.406325] Lustre: Skipped 1 previous similar message [10194.803729] Lustre: server umount lustre-OST0000 complete [10199.523555] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10199.524867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10199.539878] Lustre: Skipped 2 previous similar messages [10204.642693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10204.645219] Lustre: Skipped 2 previous similar messages [10209.765560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10211.296175] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10211.381393] Lustre: server umount lustre-MDT0000 complete [10213.385371] LustreError: 301465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784593850 with bad export cookie 15271227712910381623 [10213.388317] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10213.393553] LustreError: 301465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10213.599975] Lustre: server umount lustre-MDT0001 complete [10219.310967] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [10220.908290] Key type lgssc unregistered [10221.081204] LNet: 314723:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10221.087365] LNetError: 314723:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10221.096734] LNet: Removed LNI 192.168.201.159@tcp [10221.566305] Key type .llcrypt unregistered [10221.567683] Key type ._llcrypt unregistered [10235.749177] Key type ._llcrypt registered [10235.750431] Key type .llcrypt registered [10235.801782] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [10244.632514] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [10245.340676] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10245.363088] alg: No test for adler32 (adler32-zlib) [10246.274668] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [10246.400911] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [10248.000151] Key type lgssc registered [10248.435889] Lustre: Echo OBD driver; http://www.lustre.org/ [10264.717262] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10264.733445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10265.839526] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10265.861772] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10265.926403] Lustre: lustre-MDT0000: new disk, initializing [10265.993106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10266.006358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10267.960039] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [10273.937963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10274.020376] Lustre: 318555:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10274.057533] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10274.060078] Lustre: Skipped 1 previous similar message [10274.108834] Lustre: lustre-MDT0001: new disk, initializing [10274.172793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10274.200705] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10274.206704] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10276.557963] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [10281.737516] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10284.835523] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10288.514673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10288.703556] Lustre: lustre-OST0000: new disk, initializing [10288.706533] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10288.709809] Lustre: 320520:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10288.756532] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10291.875383] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [10296.346606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10296.359742] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10296.432662] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10297.389327] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10298.453118] Lustre: Mounted lustre-client [10298.933314] Lustre: Unmounted lustre-client [10303.459146] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10303.467082] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10303.470879] Lustre: Skipped 1 previous similar message [10308.580280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10308.583025] Lustre: Skipped 1 previous similar message [10309.473175] Lustre: server umount lustre-OST0000 complete [10313.699748] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10313.705028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10313.709538] Lustre: Skipped 2 previous similar messages [10317.367537] Lustre: server umount lustre-MDT0000 complete [10319.074389] LustreError: 320519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784593955 with bad export cookie 11377406944216403955 [10319.077963] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10319.079329] LustreError: 320519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10319.230312] Lustre: server umount lustre-MDT0001 complete [10323.892383] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [10325.140830] Key type lgssc unregistered [10351.907903] LNet: 322677:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10351.913130] LNetError: 322677:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10429.739851] LNet: Removed LNI 192.168.201.159@tcp [10430.142959] Key type .llcrypt unregistered [10430.144512] Key type ._llcrypt unregistered [10438.581727] Key type ._llcrypt registered [10438.583069] Key type .llcrypt registered [10438.641466] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [10446.947171] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [10447.439778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10447.506112] alg: No test for adler32 (adler32-zlib) [10448.435073] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [10448.557568] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [10450.160176] Key type lgssc registered [10450.718397] Lustre: Echo OBD driver; http://www.lustre.org/ [10465.422746] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 20:35:01 (1784594101) [10468.792852] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10468.817261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10469.942212] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10469.964530] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10470.004843] Lustre: lustre-MDT0000: new disk, initializing [10470.055702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10470.064713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10472.119959] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [10477.990809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10478.051425] Lustre: 325875:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10478.074337] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10478.076756] Lustre: Skipped 1 previous similar message [10478.115262] Lustre: lustre-MDT0001: new disk, initializing [10478.162206] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10478.180657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10478.189224] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10480.031839] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [10484.862474] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10487.106223] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10489.985639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10490.120915] Lustre: lustre-OST0000: new disk, initializing [10490.123202] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10490.125967] Lustre: 327843:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10490.163430] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10492.763066] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [10493.315157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10493.319200] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10493.370569] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10497.145805] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10505.985911] Lustre: server umount lustre-OST0000 complete [10508.769865] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10508.770488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10508.775735] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10508.775792] Lustre: Skipped 1 previous similar message [10513.889481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10513.892931] Lustre: Skipped 2 previous similar messages [10518.031803] Lustre: server umount lustre-MDT0000 complete [10519.710387] LustreError: 325868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784594156 with bad export cookie 9968618443656579100 [10519.713633] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10519.715302] LustreError: 325868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10529.321042] Lustre: server umount lustre-MDT0001 complete [10534.329678] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [10535.792214] Key type lgssc unregistered [10535.967621] LNet: 329749:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10535.972194] LNetError: 329749:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10535.979877] LNet: Removed LNI 192.168.201.159@tcp [10536.438668] Key type .llcrypt unregistered [10536.440497] Key type ._llcrypt unregistered [10546.915222] Key type ._llcrypt registered [10546.916385] Key type .llcrypt registered [10546.958740] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [10547.305963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10547.367684] alg: No test for adler32 (adler32-zlib) [10548.266972] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [10548.378150] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [10549.976579] Key type lgssc registered [10550.451558] Lustre: Echo OBD driver; http://www.lustre.org/ [10553.873117] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10553.881037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10555.050974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10556.603933] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [10559.824315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10559.960715] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10561.507441] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [10564.545555] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10567.132382] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10569.972962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10570.110020] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10572.357977] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [10575.451657] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10589.451553] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10590.126685] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10592.042877] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.59@tcp (stopping) [10594.787172] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10594.787994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10594.791884] Lustre: Skipped 2 previous similar messages [10594.793613] Lustre: Skipped 1 previous similar message [10597.156396] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.59@tcp (stopping) [10597.158531] Lustre: Skipped 1 previous similar message [10599.905812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10599.908891] Lustre: Skipped 2 previous similar messages [10605.026701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10605.029380] Lustre: Skipped 3 previous similar messages [10606.048188] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10606.122591] Lustre: server umount lustre-MDT0000 complete [10607.396929] LustreError: 331025:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10607.407418] LustreError: 331025:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10607.891249] LustreError: 331735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784594244 with bad export cookie 9697913411495402347 [10607.895374] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10607.896741] LustreError: 331735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10610.146841] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10610.147735] LustreError: 333490:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10612.194982] LustreError: 331025:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10614.140655] Lustre: server umount lustre-MDT0001 complete [10616.076995] Lustre: server umount lustre-OST0000 complete [10620.207084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10622.055984] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [10623.278178] Lustre: 335198:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.159@tcp [10624.167050] Lustre: server umount MGS complete [10627.787288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10628.055482] LustreError: 335770:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10628.062850] LustreError: 335770:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10628.091139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10630.002743] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [10632.994043] LustreError: 335767:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10633.901603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10634.060129] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10635.729196] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [10639.755405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10642.551528] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [10645.538375] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10646.190739] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [10648.424705] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10654.269795] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [10657.252691] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10657.256075] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10657.259234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10657.263868] Lustre: Skipped 1 previous similar message [10657.268052] Lustre: Skipped 3 previous similar messages [10661.781989] Lustre: server umount lustre-OST0000 complete [10664.929234] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10664.933391] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10669.537285] Lustre: server umount lustre-MDT0000 complete [10671.230721] LustreError: 335752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784594308 with bad export cookie 9697913411495403173 [10671.234309] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10671.238091] LustreError: 335752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10671.367774] Lustre: server umount lustre-MDT0001 complete [10675.583153] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [10676.770096] Key type lgssc unregistered [10676.920686] LNet: 339206:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10676.924281] LNetError: 339206:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10676.935964] LNet: Removed LNI 192.168.201.159@tcp [10677.337296] Key type .llcrypt unregistered [10677.338719] Key type ._llcrypt unregistered [10685.513167] Key type ._llcrypt registered [10685.514391] Key type .llcrypt registered [10685.561226] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [10693.023643] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [10693.509800] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10693.518605] alg: No test for adler32 (adler32-zlib) [10694.398734] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [10694.497416] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [10696.080504] Key type lgssc registered [10696.523583] Lustre: Echo OBD driver; http://www.lustre.org/ [10710.569462] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10710.580946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10711.668409] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10711.681421] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10711.717638] Lustre: lustre-MDT0000: new disk, initializing [10711.745323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10711.752695] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10713.292153] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [10718.054667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10718.097227] Lustre: 342339:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10718.121958] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10718.123914] Lustre: Skipped 1 previous similar message [10718.159123] Lustre: lustre-MDT0001: new disk, initializing [10718.195226] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10718.209188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10718.212993] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10719.684771] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [10723.778589] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10725.698130] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10728.185465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10728.298649] Lustre: lustre-OST0000: new disk, initializing [10728.300960] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10728.303691] Lustre: 344307:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10728.337174] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10730.465305] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [10734.607206] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10736.175563] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10737.147660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10737.151415] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10737.183299] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10737.280181] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [10738.834519] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10738.903297] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10742.243963] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10742.244100] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10742.244953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10742.247357] LustreError: Skipped 1 previous similar message [10742.252333] Lustre: Skipped 1 previous similar message [10745.691348] Lustre: server umount lustre-OST0000 complete [10747.361468] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10747.362576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10747.366767] Lustre: Skipped 1 previous similar message [10747.370889] Lustre: Skipped 2 previous similar messages [10752.481237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10752.483198] Lustre: Skipped 1 previous similar message [10753.247543] Lustre: server umount lustre-MDT0000 complete [10754.593205] LustreError: 343133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784594391 with bad export cookie 3930077058702872921 [10754.596703] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10754.598177] LustreError: 343133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10754.725064] Lustre: server umount lustre-MDT0001 complete [10757.987232] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 20:39:53 (1784594393) [10764.636707] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [10768.956723] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [10782.682898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10782.781092] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10782.793859] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10782.833367] Lustre: lustre-MDT0000: new disk, initializing [10782.868294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10782.882436] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10784.649519] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [10789.429376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10789.476554] Lustre: 349068:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10789.505201] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10789.507552] Lustre: Skipped 1 previous similar message [10789.543894] Lustre: lustre-MDT0001: new disk, initializing [10789.584300] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10789.600290] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10789.604287] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10791.284856] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [10796.059310] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10798.540938] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10801.594629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10801.761616] Lustre: lustre-OST0000: new disk, initializing [10801.764156] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10801.768373] Lustre: 351034:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10801.810509] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10804.469354] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [10808.936139] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10809.351855] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10809.356441] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10809.403682] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10824.674442] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10824.676622] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10824.678916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10824.683368] Lustre: Skipped 1 previous similar message [10826.325756] Lustre: server umount lustre-OST0000 complete [10829.793927] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10829.795649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10829.798733] Lustre: Skipped 1 previous similar message [10829.802158] Lustre: Skipped 1 previous similar message [10833.843920] Lustre: server umount lustre-MDT0000 complete [10835.426397] LustreError: 349058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784594472 with bad export cookie 3930077058702873796 [10835.428587] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10835.431423] LustreError: 349058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10835.592436] Lustre: server umount lustre-MDT0001 complete [10839.890533] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [10841.162838] Key type lgssc unregistered [10841.306464] LNet: 353181:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10841.309203] LNetError: 353181:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10841.315821] LNet: Removed LNI 192.168.201.159@tcp [10841.675621] Key type .llcrypt unregistered [10841.676939] Key type ._llcrypt unregistered [10880.845821] Key type ._llcrypt registered [10880.847026] Key type .llcrypt registered [10880.891885] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [10881.016253] Key type .llcrypt unregistered [10881.017611] Key type ._llcrypt unregistered [10912.999958] Key type ._llcrypt registered [10913.001391] Key type .llcrypt registered [10913.053475] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [10913.174512] Key type .llcrypt unregistered [10913.176291] Key type ._llcrypt unregistered [10926.510815] Key type ._llcrypt registered [10926.512114] Key type .llcrypt registered [10926.562499] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [10926.682981] Key type .llcrypt unregistered [10926.684646] Key type ._llcrypt unregistered [10941.101815] Key type ._llcrypt registered [10941.103240] Key type .llcrypt registered [10941.160959] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [10941.298940] Key type .llcrypt unregistered [10941.300368] Key type ._llcrypt unregistered [10973.339846] Key type ._llcrypt registered [10973.341442] Key type .llcrypt registered [10973.386772] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [10973.526396] Key type .llcrypt unregistered [10973.528867] Key type ._llcrypt unregistered [10986.015750] Key type ._llcrypt registered [10986.016934] Key type .llcrypt registered [10986.062493] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [10986.206495] Key type .llcrypt unregistered [10986.207970] Key type ._llcrypt unregistered [10999.957325] Key type ._llcrypt registered [10999.958560] Key type .llcrypt registered [11000.000909] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [11000.129779] Key type .llcrypt unregistered [11000.132169] Key type ._llcrypt unregistered [11013.308400] Key type ._llcrypt registered [11013.309968] Key type .llcrypt registered [11013.359989] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [11013.491237] Key type .llcrypt unregistered [11013.493874] Key type ._llcrypt unregistered [11026.646712] Key type ._llcrypt registered [11026.648167] Key type .llcrypt registered [11026.693743] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [11026.860363] Key type .llcrypt unregistered [11026.862591] Key type ._llcrypt unregistered [11039.347856] Key type ._llcrypt registered [11039.349042] Key type .llcrypt registered [11039.390454] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [11039.496434] Key type .llcrypt unregistered [11039.497865] Key type ._llcrypt unregistered [11064.118178] Key type ._llcrypt registered [11064.119740] Key type .llcrypt registered [11064.161233] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [11064.266433] Key type .llcrypt unregistered [11064.267537] Key type ._llcrypt unregistered [11075.890355] Key type ._llcrypt registered [11075.891694] Key type .llcrypt registered [11075.939244] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [11076.076298] Key type .llcrypt unregistered [11076.078332] Key type ._llcrypt unregistered [11088.218337] Key type ._llcrypt registered [11088.220030] Key type .llcrypt registered [11088.265493] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [11088.393268] Key type .llcrypt unregistered [11088.394675] Key type ._llcrypt unregistered [11109.266276] Key type ._llcrypt registered [11109.267603] Key type .llcrypt registered [11109.306781] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [11116.207698] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [11116.564517] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11116.606107] alg: No test for adler32 (adler32-zlib) [11117.481296] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [11117.600344] LNet: Added LNI 192.168.201.159@tcp [8/256/0/180] [11119.201735] Key type lgssc registered [11119.700329] Lustre: Echo OBD driver; http://www.lustre.org/ [11133.428062] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11133.433400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11134.520831] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [11134.536140] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [11134.574398] Lustre: tst250fs-MDT0000: new disk, initializing [11134.611728] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11134.623290] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11136.328949] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [11141.046081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11141.098183] Lustre: 373128:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11141.129617] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [11141.131647] Lustre: Skipped 1 previous similar message [11141.176331] Lustre: tst250fs-MDT0001: new disk, initializing [11141.226223] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11141.244852] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11141.249169] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11142.976914] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [11145.605630] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11148.238983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11148.366683] Lustre: tst250fs-OST0000: new disk, initializing [11148.368862] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [11148.372930] Lustre: 374761:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11148.408877] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11150.851455] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [11151.624362] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11151.629200] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11151.675534] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11155.646826] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11155.721776] Lustre: tst250fs-OST0001: new disk, initializing [11155.725618] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [11155.729485] Lustre: 375632:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11155.773989] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11157.964499] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_default_debug -1 all [11160.799380] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [11161.043374] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11161.050225] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11161.099297] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11171.808906] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11171.811973] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11171.817798] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [11176.419486] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [11176.423060] Lustre: tst250fs-MDT0000-lwp-OST0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11176.427533] Lustre: Skipped 1 previous similar message [11181.540582] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [11181.543090] Lustre: Skipped 6 previous similar messages [11186.144143] Lustre: tst250fs-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11186.221825] Lustre: server umount tst250fs-MDT0000 complete [11187.518412] LustreError: 375631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784594824 with bad export cookie 12193925608381917885 [11187.521897] LustreError: MGC192.168.201.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11187.522779] LustreError: 375631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11187.641263] Lustre: server umount tst250fs-MDT0001 complete [11198.983650] Lustre: server umount tst250fs-OST0000 complete [11210.668091] Lustre: server umount tst250fs-OST0001 complete [11217.141263] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [11220.136813] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [11233.017428] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 20:47:49 (1784594869) [11233.586922] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [11239.746807] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing set_hostid [11243.657962] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing load_modules_local [11256.046927] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 10875 sec ========= 20:48:11 (1784594891) [11256.719118] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 20:48:12 (1784594892) === [11258.201350] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 20:48:14 (1784594894) === [11272.873699] Lustre: DEBUG MARKER: oleg159-server.virtnet: executing unload_modules_local [11273.954408] Key type lgssc unregistered [11274.091942] LNet: 384757:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11274.095422] LNetError: 384757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11274.107431] LNet: Removed LNI 192.168.201.159@tcp [11274.448134] Key type .llcrypt unregistered [11274.449287] Key type ._llcrypt unregistered