[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 549568391 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002332] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.007269] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008025] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009015] pid_max: default: 32768 minimum: 301 [ 0.010143] LSM: Security Framework initializing [ 0.011071] Yama: becoming mindful. [ 0.012044] SELinux: Initializing. [ 0.013076] *** VALIDATE selinux *** [ 0.021773] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026168] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027163] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029102] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030115] *** VALIDATE tmpfs *** [ 0.032166] *** VALIDATE proc *** [ 0.033302] *** VALIDATE cgroup *** [ 0.034012] *** VALIDATE cgroup2 *** [ 0.035294] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037151] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039029] Spectre V2 : User space: Vulnerable [ 0.040010] Speculative Store Bypass: Vulnerable [ 0.044144] debug: unmapping init [mem 0xffffffffa9459000-0xffffffffa9460fff] [ 0.046200] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047745] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048023] ... version: 2 [ 0.049013] ... bit width: 48 [ 0.050012] ... generic registers: 4 [ 0.051015] ... value mask: 0000ffffffffffff [ 0.052016] ... max period: 00007fffffffffff [ 0.053016] ... fixed-purpose events: 3 [ 0.054012] ... event mask: 000000070000000f [ 0.055322] rcu: Hierarchical SRCU implementation. [ 0.057567] smp: Bringing up secondary CPUs ... [ 0.058625] x86: Booting SMP configuration: [ 0.059034] .... node #0, CPUs: #1 #2 #3 [ 0.062200] smp: Brought up 1 node, 4 CPUs [ 0.064013] smpboot: Max logical packages: 1 [ 0.065019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.151021] node 0 deferred pages initialised in 83ms [ 0.153145] devtmpfs: initialized [ 0.154214] x86/mm: Memory block size: 128MB [ 0.158445] gcov: version magic: 0x41383552 [ 0.160311] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.161094] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.162287] pinctrl core: initialized pinctrl subsystem [ 0.163214] [ 0.163726] ************************************************************* [ 0.164014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.165015] ** ** [ 0.166015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.167013] ** ** [ 0.168014] ** This means that this kernel is built to expose internal ** [ 0.169011] ** IOMMU data structures, which may compromise security on ** [ 0.170011] ** your system. ** [ 0.171015] ** ** [ 0.172011] ** If you see this message and you are not debugging the ** [ 0.173010] ** kernel, report this immediately to your vendor! ** [ 0.174015] ** ** [ 0.175010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.176011] ************************************************************* [ 0.177813] NET: Registered protocol family 16 [ 0.178469] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.179088] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.180071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.182075] cpuidle: using governor menu [ 0.183666] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.186540] PCI: Using configuration type 1 for base access [ 0.188130] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.198321] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.201018] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.204121] cryptd: max_cpu_qlen set to 1000 [ 0.207046] ACPI: Added _OSI(Module Device) [ 0.208011] ACPI: Added _OSI(Processor Device) [ 0.209016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.210016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.214799] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.217347] ACPI: Interpreter enabled [ 0.218067] ACPI: PM: (supports S0 S3 S4 S5) [ 0.219015] ACPI: Using IOAPIC for interrupt routing [ 0.220086] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.221423] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.231113] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.232051] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.233018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.234082] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.236447] acpiphp: Slot [2] registered [ 0.237136] acpiphp: Slot [5] registered [ 0.238098] acpiphp: Slot [6] registered [ 0.239113] acpiphp: Slot [7] registered [ 0.240178] acpiphp: Slot [8] registered [ 0.241165] acpiphp: Slot [9] registered [ 0.242128] acpiphp: Slot [10] registered [ 0.243131] acpiphp: Slot [3] registered [ 0.244101] acpiphp: Slot [4] registered [ 0.245102] acpiphp: Slot [11] registered [ 0.246170] acpiphp: Slot [12] registered [ 0.247099] acpiphp: Slot [13] registered [ 0.248093] acpiphp: Slot [14] registered [ 0.249086] acpiphp: Slot [15] registered [ 0.250089] acpiphp: Slot [16] registered [ 0.251096] acpiphp: Slot [17] registered [ 0.252082] acpiphp: Slot [18] registered [ 0.253119] acpiphp: Slot [19] registered [ 0.254088] acpiphp: Slot [20] registered [ 0.255155] acpiphp: Slot [21] registered [ 0.256138] acpiphp: Slot [22] registered [ 0.257095] acpiphp: Slot [23] registered [ 0.258195] acpiphp: Slot [24] registered [ 0.259119] acpiphp: Slot [25] registered [ 0.260085] acpiphp: Slot [26] registered [ 0.261092] acpiphp: Slot [27] registered [ 0.262125] acpiphp: Slot [28] registered [ 0.263081] acpiphp: Slot [29] registered [ 0.264080] acpiphp: Slot [30] registered [ 0.265107] acpiphp: Slot [31] registered [ 0.266074] PCI host bridge to bus 0000:00 [ 0.267018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.268019] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.269027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.270026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.271028] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.272029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.273196] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.274595] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.278117] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.286015] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.289050] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.292020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.294012] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.295012] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.297474] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.299497] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.301027] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.303854] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.309026] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.320017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.325014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.332755] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.342023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.346019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.357018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.366172] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.372015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.378015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.389022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.397012] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.401016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.405013] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.418014] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.425773] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.431015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.435012] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.453025] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.467212] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.473021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.479020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.495033] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.508165] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.518028] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.529023] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.552020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.575734] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.577352] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.580362] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.582388] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.584196] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.588359] iommu: Default domain type: Passthrough [ 0.590453] SCSI subsystem initialized [ 0.592113] ACPI: bus type USB registered [ 0.593124] usbcore: registered new interface driver usbfs [ 0.595094] usbcore: registered new interface driver hub [ 0.597107] usbcore: registered new device driver usb [ 0.599174] pps_core: LinuxPPS API ver. 1 registered [ 0.600010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.603084] PTP clock support registered [ 0.605104] EDAC MC: Ver: 3.0.0 [ 0.606482] PCI: Using ACPI for IRQ routing [ 0.608878] NetLabel: Initializing [ 0.610014] NetLabel: domain hash size = 128 [ 0.611007] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.612085] NetLabel: unlabeled traffic allowed by default [ 0.614116] vgaarb: loaded [ 0.616278] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.617018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.623389] clocksource: Switched to clocksource kvm-clock [ 0.723434] VFS: Disk quotas dquot_6.6.0 [ 0.724557] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.726181] *** VALIDATE ramfs *** [ 0.726903] *** VALIDATE hugetlbfs *** [ 0.727863] pnp: PnP ACPI init [ 0.729483] pnp: PnP ACPI: found 6 devices [ 0.743904] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.745950] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.747276] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.748608] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.750022] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.751362] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.753182] NET: Registered protocol family 2 [ 0.754856] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.758172] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.760416] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.763889] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.766329] TCP: Hash tables configured (established 65536 bind 65536) [ 0.768966] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.771489] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.773749] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.776219] NET: Registered protocol family 1 [ 0.778795] RPC: Registered named UNIX socket transport module. [ 0.780369] RPC: Registered udp transport module. [ 0.781480] RPC: Registered tcp transport module. [ 0.782475] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.784049] NET: Registered protocol family 44 [ 0.785303] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.788018] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.791722] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.795722] PCI: CLS 0 bytes, default 64 [ 0.799971] Unpacking initramfs... [ 2.158508] debug: unmapping init [mem 0xffff99377cc54000-0xffff99377ffbffff] [ 2.162308] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.164532] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.167475] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.635828] Initialise system trusted keyrings [ 2.637021] Key type blacklist registered [ 2.638157] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.647367] zbud: loaded [ 2.651665] *** VALIDATE nfs *** [ 2.653303] *** VALIDATE nfs4 *** [ 2.654844] pstore: using deflate compression [ 2.657566] Platform Keyring initialized [ 2.740593] NET: Registered protocol family 38 [ 2.741871] Key type asymmetric registered [ 2.742968] Asymmetric key parser 'x509' registered [ 2.744264] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.746274] io scheduler mq-deadline registered [ 2.747407] io scheduler kyber registered [ 2.748533] io scheduler bfq registered [ 2.749706] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.751463] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.753193] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.754913] ACPI: Power Button [PWRF] [ 2.758329] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.762794] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.775309] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.779800] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.793631] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.818025] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.845458] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.849771] Non-volatile memory driver v1.3 [ 2.850842] Linux agpgart interface v0.103 [ 2.874843] virtio_blk virtio1: [vda] 145880 512-byte logical blocks (74.7 MB/71.2 MiB) [ 2.876648] vda: detected capacity change from 0 to 74690560 [ 2.888367] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.890499] vdb: detected capacity change from 0 to 1073741824 [ 2.904364] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.906268] vdc: detected capacity change from 0 to 2621440000 [ 2.918294] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.921282] vdd: detected capacity change from 0 to 2621440000 [ 2.935385] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.937302] vde: detected capacity change from 0 to 4294967296 [ 2.951633] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.953465] vdf: detected capacity change from 0 to 4294967296 [ 2.957731] libphy: Fixed MDIO Bus: probed [ 2.964242] usbcore: registered new interface driver usbserial_generic [ 2.966119] usbserial: USB Serial support registered for generic [ 2.968045] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.971724] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.973214] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.974987] mousedev: PS/2 mouse device common for all mice [ 2.978295] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.981828] rtc_cmos 00:05: RTC can wake from S4 [ 2.986373] rtc_cmos 00:05: registered as rtc0 [ 2.986544] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.987443] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.995714] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.996571] intel_pstate: CPU model not supported [ 3.006592] hid: raw HID events driver (C) Jiri Kosina [ 3.009020] usbcore: registered new interface driver usbhid [ 3.011057] usbhid: USB HID core driver [ 3.012648] drop_monitor: Initializing network drop monitor service [ 3.014952] Initializing XFRM netlink socket [ 3.016976] NET: Registered protocol family 10 [ 3.019593] Segment Routing with IPv6 [ 3.020415] NET: Registered protocol family 17 [ 3.022138] mpls_gso: MPLS GSO support [ 3.027719] RAS: Correctable Errors collector initialized. [ 3.029697] AVX version of gcm_enc/dec engaged. [ 3.030671] AES CTR mode by8 optimization enabled [ 3.092202] sched_clock: Marking stable (3092183444, 0)->(4053343085, -961159641) [ 3.094517] registered taskstats version 1 [ 3.097249] Loading compiled-in X.509 certificates [ 3.100365] zswap: loaded using pool lzo/zbud [ 3.124584] Key type big_key registered [ 3.138443] Key type encrypted registered [ 3.140507] ima: No TPM chip found, activating TPM-bypass! [ 3.143342] ima: Allocated hash algorithm: sha1 [ 3.144802] ima: No architecture policies found [ 3.146322] evm: Initialising EVM extended attributes: [ 3.148476] evm: security.selinux [ 3.150143] evm: security.ima [ 3.151277] evm: security.capability [ 3.152970] evm: HMAC attrs: 0x1 [ 3.155987] rtc_cmos 00:05: setting system clock to 2026-07-25 00:03:59 UTC (1784937839) [ 3.164320] debug: unmapping init [mem 0xffffffffaa403000-0xffffffffaa5fffff] [ 3.167596] debug: unmapping init [mem 0xffffffffa9182000-0xffffffffa9458fff] [ 3.180218] Write protecting the kernel read-only data: 28672k [ 3.183582] debug: unmapping init [mem 0xffffffffa7803000-0xffffffffa79fffff] [ 3.186055] debug: unmapping init [mem 0xffffffffa8114000-0xffffffffa81fffff] [ 3.224675] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.233400] systemd[1]: Detected virtualization kvm. [ 3.235437] systemd[1]: Detected architecture x86-64. [ 3.237378] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.263774] systemd[1]: No hostname configured. [ 3.265566] systemd[1]: Set hostname to . [ 3.267743] random: systemd: uninitialized urandom read (16 bytes read) [ 3.270125] systemd[1]: Initializing machine ID from random generator. [ 3.321593] random: ln: uninitialized urandom read (6 bytes read) [ 3.399725] random: systemd: uninitialized urandom read (16 bytes read) [ 3.402813] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.408320] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.413899] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local File Systems. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.065066] device-mapper: uevent: version 1.0.3 [ 4.067092] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. [ 4.409535] random: fast init done Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.861925] virtio_net virtio0 ens2: renamed from eth0 [ 5.020189] scsi host0: ata_piix [ 5.090955] scsi host1: ata_piix [ 5.092381] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.094312] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.770529] dracut-initqueue[582]: RTNETLINK answers: File exists [ 9.658439] random: crng init done [ 9.659607] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.158674] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Timers. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Sockets. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.455143] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.783157] SELinux: Disabled at runtime. [ 11.852718] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.864248] systemd[1]: Detected virtualization kvm. [ 11.866631] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.456440] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.461124] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.469847] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.473910] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.477761] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.488805] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.494846] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Reached target rpc_pipefs.target. Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ 12.672040] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting POSIX Message Queue File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd Root File System. Mounting Huge Pages File System... Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.262275] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.942076] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.966644] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.389767] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.414629] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (6s / no limit)[ 19.244983] Key type dns_resolver registered [** ] A start job is running for Configur…-only root support (7s / no limit) [*** ] A start job is running for Configur…-only root support (7s / no limit)[ 20.527529] NFS: Registering the id_resolver key type [ 20.530354] Key type id_resolver registered [ 20.532304] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started Login Service. [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg106-server login: [ 34.580009] hrtimer: interrupt took 19019251 ns [ 82.067766] libcfs: loading out-of-tree module taints kernel. [ 82.105784] Key type ._llcrypt registered [ 82.115863] Key type .llcrypt registered [ 82.207386] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 99.792649] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 101.231247] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 101.247342] alg: No test for adler32 (adler32-zlib) [ 102.623959] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 103.802222] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 105.607170] Key type lgssc registered [ 107.442382] Lustre: Echo OBD driver; http://www.lustre.org/ [ 125.094566] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 166.727615] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 181.713196] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 181.768257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 182.943631] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 182.965817] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 183.083329] Lustre: lustre-MDT0000: new disk, initializing [ 183.146668] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 183.166363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 187.700106] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 202.568182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 202.644281] Lustre: 6518:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 202.665344] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 202.668574] Lustre: Skipped 1 previous similar message [ 202.756399] Lustre: lustre-MDT0001: new disk, initializing [ 202.813320] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 202.846496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 202.864499] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 208.255311] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 214.034802] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 225.848199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 226.281280] Lustre: lustre-OST0000: new disk, initializing [ 226.286341] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 226.302618] Lustre: 8454:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 226.444027] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 234.054684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 234.065372] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 234.171743] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 234.837735] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 252.246467] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 252.473475] Lustre: lustre-OST0001: new disk, initializing [ 252.479576] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 252.490248] Lustre: 9526:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 252.611502] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 258.679832] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 261.704675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 261.722501] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 261.804197] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 270.334773] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 276.398436] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 282.647225] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing check_logdir /tmp/testlogs/ [ 288.517177] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing yml_node [ 292.881295] Lustre: DEBUG MARKER: Client: 2.17.54.203 [ 295.796704] Lustre: DEBUG MARKER: MDS: 2.17.54.203 [ 298.398494] Lustre: DEBUG MARKER: OSS: 2.17.54.203 [ 300.174945] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Jul 24 20:08:54 EDT 2026 [ 319.518325] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 320.939053] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 338.407917] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 338.409847] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 338.424322] Lustre: Skipped 1 previous similar message [ 343.656186] Lustre: server umount lustre-MDT0000 complete [ 347.258466] LustreError: 6508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784938183 with bad export cookie 13054764557222291184 [ 347.260833] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 347.279994] LustreError: 6508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 348.641749] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 348.643149] LustreError: 12276:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 348.643748] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.106@tcp (stopping) [ 348.643753] Lustre: Skipped 3 previous similar messages [ 348.662478] Lustre: Skipped 3 previous similar messages [ 348.695205] LustreError: 12276:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 6 previous similar messages [ 352.674727] LustreError: 6525:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 352.684965] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.106@tcp (stopping) [ 352.708436] Lustre: Skipped 2 previous similar messages [ 352.716719] LustreError: 6525:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 353.719445] Lustre: server umount lustre-MDT0001 complete [ 357.483884] Lustre: server umount lustre-OST0000 complete [ 361.147734] Lustre: server umount lustre-OST0001 complete [ 367.617686] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 377.233810] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 416.015210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 416.180292] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 416.203596] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 416.256608] Lustre: lustre-MDT0000: new disk, initializing [ 416.314093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 416.332923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 421.943895] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 433.344816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 433.476616] Lustre: 15088:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 433.526750] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 433.532020] Lustre: Skipped 1 previous similar message [ 433.610956] Lustre: lustre-MDT0001: new disk, initializing [ 433.806590] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 433.856129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 433.870333] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 438.722728] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 449.256487] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 455.565603] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 462.514426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 462.791533] Lustre: lustre-OST0000: new disk, initializing [ 462.795508] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 462.803117] Lustre: 17060:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 462.895339] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 464.425725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 464.445951] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 464.568327] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 469.437230] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 480.783422] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 486.270547] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 486.492564] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 492.024826] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 492.238846] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 495.593235] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.201.106@tcp failed: rc = -107 [ 495.605634] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 495.638223] Lustre: lustre-OST0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 495.649930] Lustre: Skipped 1 previous similar message [ 495.668325] Lustre: Skipped 1 previous similar message [ 500.681100] Lustre: server umount lustre-OST0000 complete [ 505.312879] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.201.106@tcp failed: rc = -107 [ 505.323049] LustreError: Skipped 1 previous similar message [ 505.326424] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 505.335838] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 510.124719] Lustre: server umount lustre-MDT0000 complete [ 510.945618] LustreError: 15101:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 510.963081] LustreError: 15101:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 513.480968] LustreError: 15082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784938349 with bad export cookie 13054764557222292717 [ 513.488898] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 513.493575] LustreError: 15082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 513.965724] Lustre: server umount lustre-MDT0001 complete [ 519.738941] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 20:12:34 (1784938354) [ 538.347986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 538.678815] LustreError: 19934:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 538.736243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 543.361130] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 544.236817] LustreError: 19935:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 553.263913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 558.529090] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 567.479739] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 573.847325] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 576.275582] Lustre: Failing over lustre-MDT0000 [ 576.638716] Lustre: server umount lustre-MDT0000 complete [ 579.364541] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 579.368352] LustreError: 19935:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 579.374521] Lustre: Skipped 2 previous similar messages [ 579.410378] LustreError: 19935:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 595.209232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 595.338260] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 599.657065] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 599.680565] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.201.106@tcp (at 192.168.201.106@tcp) [ 599.705934] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 600.613780] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 613.469934] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 620.535249] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 620.535670] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.201.106@tcp failed: rc = -107 [ 620.569950] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 620.569963] Lustre: Skipped 1 previous similar message [ 632.287306] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 632.657362] Lustre: server umount lustre-MDT0000 complete [ 634.848726] LustreError: 19935:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 634.873870] LustreError: 19935:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 7 previous similar messages [ 637.740312] LustreError: 19914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784938474 with bad export cookie 13054764557222294131 [ 637.743816] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 637.749391] LustreError: 19914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 638.060409] Lustre: server umount lustre-MDT0001 complete [ 649.492798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 650.000344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 650.006438] Lustre: Skipped 2 previous similar messages [ 655.055927] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 665.789669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 672.601156] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 682.682627] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 689.728305] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 692.379528] Lustre: Failing over lustre-MDT0000 [ 692.712713] Lustre: server umount lustre-MDT0000 complete [ 696.295462] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 696.301731] LustreError: 23272:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 696.309717] Lustre: Skipped 2 previous similar messages [ 696.344133] LustreError: 23272:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 5 previous similar messages [ 711.883200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 712.106588] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 712.378145] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 716.744101] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 717.793491] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 717.809629] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.201.106@tcp (at 192.168.201.106@tcp) [ 717.829058] Lustre: Skipped 1 previous similar message [ 717.847605] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 728.514400] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 736.355172] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 736.393173] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 736.409054] Lustre: Skipped 6 previous similar messages [ 739.765908] Lustre: server umount lustre-MDT0000 complete [ 744.991407] LustreError: 23254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784938581 with bad export cookie 13054764557222295069 [ 745.016689] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 745.361229] Lustre: server umount lustre-MDT0001 complete [ 757.066965] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 20:16:30 (1784938590) [ 775.240118] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 790.472897] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 835.556563] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 849.090352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 849.452286] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 849.497035] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 849.586992] Lustre: lustre-MDT0000: new disk, initializing [ 849.654760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 849.662813] Lustre: Skipped 2 previous similar messages [ 849.704280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 855.156603] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 866.750730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 866.900414] Lustre: 29669:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 867.133958] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 872.124430] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 883.122798] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 890.366044] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 898.987032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 899.377796] Lustre: 31637:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 901.132353] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 901.178841] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 901.288423] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 901.318859] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 905.621694] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 916.490593] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 923.458255] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 952.324586] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.201.106@tcp failed: rc = -107 [ 952.341120] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 952.364824] Lustre: Skipped 1 previous similar message [ 952.382821] Lustre: lustre-OST0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 955.491391] Lustre: server umount lustre-OST0000 complete [ 966.922712] Lustre: server umount lustre-MDT0000 complete [ 968.162841] LustreError: 29680:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 968.179513] LustreError: 29680:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 9 previous similar messages [ 971.101971] LustreError: 29659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784938807 with bad export cookie 13054764557222295475 [ 971.111420] LustreError: 29659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 971.116211] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 971.553631] Lustre: server umount lustre-MDT0001 complete [ 983.893779] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 987.221144] Key type lgssc unregistered [ 987.605377] LNet: 33618:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 987.630852] LNetError: 33618:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 987.676548] LNet: Removed LNI 192.168.201.106@tcp [ 988.710265] Key type .llcrypt unregistered [ 988.713975] Key type ._llcrypt unregistered [ 1015.002321] Key type ._llcrypt registered [ 1015.004840] Key type .llcrypt registered [ 1015.079594] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 1016.348038] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1016.357749] alg: No test for adler32 (adler32-zlib) [ 1017.386941] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 1017.614955] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 1019.263190] Key type lgssc registered [ 1020.650087] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1027.680560] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1027.693680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1029.068494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1033.779305] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1039.331502] LustreError: 35034:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1039.362399] LustreError: 35034:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 1042.108039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1042.539386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1047.385946] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1055.302137] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1062.769429] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1072.088101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1072.475761] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1080.571614] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1086.921362] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1090.304872] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1106.652623] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 20:22:20 (1784938940) [ 1109.073404] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1111.448501] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1119.792309] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 20:22:33 (1784938953) [ 1129.837656] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 20:22:43 (1784938963) [ 1130.950322] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1141.667461] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1141.671931] Lustre: Skipped 2 previous similar messages [ 1144.060207] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 20:22:57 (1784938977) [ 1146.280635] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1146.285665] Lustre: Skipped 2 previous similar messages [ 1154.671297] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 1154.678611] Lustre: Skipped 1 previous similar message [ 1170.764236] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1170.777855] Lustre: Skipped 5 previous similar messages [ 1174.167891] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 20:23:28 (1784939008) [ 1177.560758] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1181.517373] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1190.604864] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 20:23:45 (1784939025) [ 1205.724844] Lustre: Setting parameter general.jobid_name=jobid_name=jobid_name=%e.root in log params [ 1205.736423] Lustre: Skipped 7 previous similar messages [ 1208.245562] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 20:24:02 (1784939042) [ 1228.261601] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.201.106@tcp failed: rc = -107 [ 1228.280544] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1228.287289] Lustre: lustre-OST0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 1228.317278] Lustre: Skipped 1 previous similar message [ 1228.346742] Lustre: Skipped 1 previous similar message [ 1233.390905] LustreError: 37466:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1233.420061] LustreError: 37466:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 1233.705963] Lustre: server umount lustre-OST0000 complete [ 1238.500526] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1238.514197] Lustre: Skipped 1 previous similar message [ 1238.518685] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 1238.524453] Lustre: Skipped 1 previous similar message [ 1243.640077] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 1243.659660] Lustre: Skipped 1 previous similar message [ 1244.565679] Lustre: server umount lustre-MDT0000 complete [ 1248.743657] LustreError: 35762:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1248.787101] LustreError: 35762:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 1249.425283] LustreError: 38594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784939085 with bad export cookie 17630686932905389870 [ 1249.432198] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1249.434732] LustreError: 38594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1249.782862] Lustre: server umount lustre-MDT0001 complete [ 1263.484287] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 1265.458162] Key type lgssc unregistered [ 1265.649257] LNet: 42892:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1265.664749] LNetError: 42892:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1266.727959] LNet: Removed LNI 192.168.201.106@tcp [ 1267.895134] Key type .llcrypt unregistered [ 1267.910858] Key type ._llcrypt unregistered [ 1287.476984] Key type ._llcrypt registered [ 1287.483094] Key type .llcrypt registered [ 1287.628053] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 1306.979851] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 1308.330028] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1308.387631] alg: No test for adler32 (adler32-zlib) [ 1309.455797] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 1309.612637] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 1311.249428] Key type lgssc registered [ 1312.204656] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1351.789579] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1351.848781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1353.175992] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1353.239548] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1353.364624] Lustre: lustre-MDT0000: new disk, initializing [ 1353.548827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1353.581908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1358.938415] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1370.852453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1371.008656] Lustre: 46052:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1371.055487] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1371.061310] Lustre: Skipped 1 previous similar message [ 1371.143775] Lustre: lustre-MDT0001: new disk, initializing [ 1371.232066] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1371.275510] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1371.292800] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1376.369063] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1387.982308] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1396.523587] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1404.356853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1404.763731] Lustre: lustre-OST0000: new disk, initializing [ 1404.768482] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1404.774052] Lustre: 48023:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1404.892854] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1411.572731] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1413.711426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1413.721988] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1413.823673] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1421.093687] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1425.378765] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1425.545618] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1430.016534] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1430.238861] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1434.601078] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.201.106@tcp failed: rc = -107 [ 1434.607226] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1434.616626] Lustre: lustre-OST0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 1434.648383] Lustre: Skipped 1 previous similar message [ 1438.513950] Lustre: server umount lustre-OST0000 complete [ 1444.844882] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1444.851863] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 1444.883638] Lustre: Skipped 1 previous similar message [ 1444.893173] Lustre: Skipped 1 previous similar message [ 1451.546389] Lustre: server umount lustre-MDT0000 complete [ 1455.112131] LustreError: 46059:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1455.160207] LustreError: 46059:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 4 previous similar messages [ 1456.132394] LustreError: 46045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784939292 with bad export cookie 10684876875709625492 [ 1456.136110] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1456.154099] LustreError: 46045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1456.761483] Lustre: server umount lustre-MDT0001 complete [ 1468.247747] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 20:28:22 (1784939302) [ 1474.690915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1475.308834] LustreError: 50483:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1475.359667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1480.014594] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1480.676213] LustreError: 50484:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1484.781436] LustreError: 50483:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1488.856633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1489.135726] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1494.210259] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1502.444040] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1508.999901] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1516.903129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1517.324041] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1525.283512] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1532.878114] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1538.778392] LustreError: 53014:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1539.902816] Lustre: Modifying parameter general.timeout=1 in log params [ 1541.710700] Lustre: Modifying parameter general.timeout=3 in log params [ 1541.717801] Lustre: Skipped 1 previous similar message [ 1544.087340] Lustre: Modifying parameter general.timeout=6 in log params [ 1544.090848] Lustre: Skipped 2 previous similar messages [ 1548.497388] Lustre: Modifying parameter general.timeout=12 in log params [ 1548.500425] Lustre: Skipped 5 previous similar messages [ 1556.647451] Lustre: Modifying parameter general.timeout=23 in log params [ 1556.650164] Lustre: Skipped 10 previous similar messages [ 1572.866896] Lustre: Modifying parameter general.timeout=42 in log params [ 1572.873266] Lustre: Skipped 18 previous similar messages [ 1605.571663] Lustre: Modifying parameter general.timeout=77 in log params [ 1605.575049] Lustre: Skipped 34 previous similar messages [ 1664.167095] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 20:31:38 (1784939498) [ 1666.864686] Lustre: 59951:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1668.856939] Lustre: 60063:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1669.785599] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 in log params [ 1669.795530] Lustre: Skipped 56 previous similar messages [ 1671.061587] Lustre: 60162:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1675.872703] Lustre: 60359:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1675.880311] Lustre: 60359:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1705.567958] Lustre: 61833:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1705.574764] Lustre: 61833:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1705.598074] LustreError: 61833:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1710.208966] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 20:32:24 (1784939544) [ 1723.873101] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1723.876579] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 1723.900212] Lustre: Skipped 1 previous similar message [ 1723.912145] Lustre: Skipped 1 previous similar message [ 1728.995246] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 1729.006100] Lustre: Skipped 4 previous similar messages [ 1730.129515] Lustre: server umount lustre-MDT0000 complete [ 1733.989582] LustreError: 54398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784939570 with bad export cookie 10684876875709626367 [ 1733.992843] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1734.010290] LustreError: 54398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1734.112287] LustreError: 51723:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1734.114392] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1734.142036] Lustre: Skipped 1 previous similar message [ 1734.148243] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.106@tcp (stopping) [ 1734.321835] Lustre: server umount lustre-MDT0001 complete [ 1740.570689] Lustre: server umount lustre-OST0000 complete [ 1766.851177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1767.123173] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1767.302827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1771.578247] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1780.410541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1780.526838] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1780.570912] Lustre: 64363:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1784.759878] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1791.366544] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1796.637267] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1803.439459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1803.559206] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1803.770747] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1803.780708] Lustre: Skipped 1 previous similar message [ 1811.181234] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1819.046550] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1821.097194] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1821.104626] Lustre: Skipped 19 previous similar messages [ 1821.168449] Lustre: 66930:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1828.837688] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.201.106@tcp failed: rc = -107 [ 1828.839780] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1828.844949] Lustre: lustre-OST0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 1828.851170] LustreError: Skipped 2 previous similar messages [ 1828.875734] Lustre: Skipped 1 previous similar message [ 1833.965101] LustreError: 66212:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1834.046652] Lustre: server umount lustre-OST0000 complete [ 1837.026150] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.201.106@tcp failed: rc = -107 [ 1837.038551] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 1849.312986] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 1849.321668] Lustre: Skipped 7 previous similar messages [ 1851.359688] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1851.634751] Lustre: server umount lustre-MDT0000 complete [ 1854.433816] LustreError: 65117:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1854.456077] LustreError: 65117:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 1855.331504] LustreError: 64358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784939691 with bad export cookie 10684876875709629384 [ 1855.343089] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1855.349434] LustreError: 64358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1855.899807] Lustre: server umount lustre-MDT0001 complete [ 1869.449562] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 1872.884798] Key type lgssc unregistered [ 1873.252579] LNet: 68136:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1873.268536] LNetError: 68136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1873.290169] LNet: Removed LNI 192.168.201.106@tcp [ 1874.711184] Key type .llcrypt unregistered [ 1874.722514] Key type ._llcrypt unregistered [ 1904.224986] Key type ._llcrypt registered [ 1904.227645] Key type .llcrypt registered [ 1904.324627] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 1905.735519] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1905.830108] alg: No test for adler32 (adler32-zlib) [ 1906.938969] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 1907.168751] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 1908.847185] Key type lgssc registered [ 1910.454197] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1919.773934] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1919.812831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1921.542945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1928.449232] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1930.731897] LustreError: 69553:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1930.759871] LustreError: 69553:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 1935.848280] LustreError: 69554:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1938.823372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1939.236654] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1944.293292] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1952.546711] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1958.937075] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1965.644384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1966.294275] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1972.029349] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1979.414627] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1991.200890] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2004.454335] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2004.473733] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 2004.487809] Lustre: Skipped 1 previous similar message [ 2004.501391] Lustre: Skipped 2 previous similar messages [ 2007.681659] Lustre: server umount lustre-MDT0000 complete [ 2009.569858] LustreError: 69553:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2009.593315] LustreError: 69553:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 2010.408390] LustreError: 71404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784939846 with bad export cookie 6809484903920753635 [ 2010.419807] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2010.429889] LustreError: 71404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2010.728620] Lustre: server umount lustre-MDT0001 complete [ 2024.306760] Lustre: server umount lustre-OST0000 complete [ 2051.551540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2051.872986] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2051.931961] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2052.215161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2057.414826] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2066.947085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2067.111784] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2067.158810] Lustre: 74424:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2067.839584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2072.339969] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2079.618419] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2085.957593] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2093.433643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2093.699223] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2094.305332] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2100.476081] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2108.815508] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2110.352236] Lustre: 76999:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2112.043525] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2120.678407] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.201.106@tcp failed: rc = -107 [ 2120.685210] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2120.699879] LustreError: Skipped 1 previous similar message [ 2120.706921] Lustre: Skipped 1 previous similar message [ 2120.709284] Lustre: lustre-OST0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 2123.329066] Lustre: server umount lustre-OST0000 complete [ 2128.883440] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.201.106@tcp failed: rc = -107 [ 2128.894637] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2128.908210] Lustre: Skipped 1 previous similar message [ 2128.917385] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 2128.922649] Lustre: Skipped 1 previous similar message [ 2130.923666] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2130.956922] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 2130.972485] Lustre: Skipped 1 previous similar message [ 2133.736490] Lustre: server umount lustre-MDT0000 complete [ 2136.033836] LustreError: 75563:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2136.052961] LustreError: 75563:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 2137.645329] LustreError: 74415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784939973 with bad export cookie 6809484903920754482 [ 2137.655253] LustreError: 74415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2137.656391] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2138.001288] Lustre: server umount lustre-MDT0001 complete [ 2149.894750] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 2152.739219] Key type lgssc unregistered [ 2153.002606] LNet: 78209:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2153.009165] LNetError: 78209:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2154.026715] LNet: Removed LNI 192.168.201.106@tcp [ 2154.918703] Key type .llcrypt unregistered [ 2154.921115] Key type ._llcrypt unregistered [ 2182.785817] Key type ._llcrypt registered [ 2182.789419] Key type .llcrypt registered [ 2182.881249] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 2184.036527] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2184.097811] alg: No test for adler32 (adler32-zlib) [ 2185.106194] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 2185.415226] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 2187.087197] Key type lgssc registered [ 2187.950318] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2194.512926] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2194.537672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2196.219359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2201.888773] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2205.703298] LustreError: 79621:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2205.735949] LustreError: 79621:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 2210.786893] LustreError: 79622:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2212.950798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2214.490977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2221.198849] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2231.891576] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2239.487823] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2246.574332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2247.755452] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2254.276968] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2263.153535] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2274.705553] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2275.707879] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2276.711248] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2279.150234] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2279.155911] Lustre: Skipped 2 previous similar messages [ 2283.829847] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2283.835345] Lustre: Skipped 4 previous similar messages [ 2292.586637] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2292.590658] Lustre: Skipped 10 previous similar messages [ 2308.720771] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2308.734551] Lustre: Skipped 18 previous similar messages [ 2340.968927] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2340.975878] Lustre: Skipped 32 previous similar messages [ 2376.591716] Lustre: 87728:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2379.746669] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 20:43:33 (1784940213) [ 2381.364762] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2383.479431] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 20:43:37 (1784940217) [ 2388.984364] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2388.996688] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.201.106@tcp failed: rc = -107 [ 2389.014791] Lustre: lustre-OST0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 2394.093171] Lustre: lustre-OST0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 2394.108166] Lustre: Skipped 1 previous similar message [ 2394.423477] Lustre: server umount lustre-OST0000 complete [ 2403.312820] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.201.106@tcp failed: rc = -107 [ 2403.330781] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2403.376684] Lustre: Skipped 1 previous similar message [ 2403.403331] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 2404.347986] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2405.680904] Lustre: server umount lustre-MDT0000 complete [ 2409.452159] LustreError: 79621:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2409.500198] LustreError: 79621:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 2411.071959] LustreError: 79604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784940247 with bad export cookie 7255387791298510104 [ 2411.072966] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2411.087648] LustreError: 79604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2411.512125] Lustre: server umount lustre-MDT0001 complete [ 2425.151113] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 2428.977800] Key type lgssc unregistered [ 2429.314828] LNet: 88978:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2429.323093] LNetError: 88978:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2429.341497] LNet: Removed LNI 192.168.201.106@tcp [ 2430.286566] Key type .llcrypt unregistered [ 2430.290760] Key type ._llcrypt unregistered [ 2434.893678] Key type ._llcrypt registered [ 2434.895734] Key type .llcrypt registered [ 2435.005595] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2439.688563] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules [ 2441.017341] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2441.071352] alg: No test for adler32 (adler32-zlib) [ 2442.148490] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 2442.374509] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 2444.079219] Key type lgssc registered [ 2445.195920] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2446.011032] LustreError: 90181:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2464.560600] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 2468.260493] LustreError: 90181:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2475.569071] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2475.587126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2477.115328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2481.603523] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2487.277875] LustreError: 91184:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2487.304369] LustreError: 91184:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 2491.206580] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 20:45:25 (1784940325) [ 2492.388150] LustreError: 91185:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2496.202638] LustreError: 91216:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2496.218461] LustreError: 91216:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2496.235545] LustreError: 91216:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 2496.540918] Lustre: server umount lustre-MDT0000 complete [ 2511.837531] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 2514.579897] Key type lgssc unregistered [ 2514.982198] LNet: 92709:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2514.990051] LNetError: 92709:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2516.080198] LNet: Removed LNI 192.168.201.106@tcp [ 2517.204766] Key type .llcrypt unregistered [ 2517.206886] Key type ._llcrypt unregistered [ 2539.634256] Key type ._llcrypt registered [ 2539.637506] Key type .llcrypt registered [ 2539.765537] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 2541.396882] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2541.476399] alg: No test for adler32 (adler32-zlib) [ 2542.558780] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 2542.803414] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 2544.471444] Key type lgssc registered [ 2545.718218] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2554.267643] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2554.311229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2555.973672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2560.770192] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2566.119682] LustreError: 93930:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2566.147670] LustreError: 93930:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 2570.431890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2571.176700] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2576.544630] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2585.455192] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2593.254884] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2601.303470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2601.885495] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2608.416624] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2616.123254] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2686.455528] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 20:48:40 (1784940520) [ 2688.106613] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2689.985461] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 20:48:44 (1784940524) [ 2698.213877] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2698.229310] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 2698.238096] Lustre: Skipped 2 previous similar messages [ 2700.604795] Lustre: server umount lustre-MDT0000 complete [ 2703.337163] LustreError: 93925:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2703.363742] LustreError: 93925:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 2704.274588] LustreError: 93912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784940540 with bad export cookie 17097179101408856870 [ 2704.283708] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2704.285311] LustreError: 93912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2704.699755] Lustre: server umount lustre-MDT0001 complete [ 2719.228647] Lustre: server umount lustre-OST0000 complete [ 2731.816034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2732.349353] LustreError: 98253:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2732.551456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2737.657994] LustreError: 98254:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2738.194964] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2741.742532] LustreError: 98253:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2748.315531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2749.174579] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2755.647179] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2764.932026] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2771.062738] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2785.525264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2785.806687] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2785.809818] LustreError: 98241:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2785.828624] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2785.832499] LustreError: 100338:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2785.836271] LustreError: 100338:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2785.838632] LustreError: 100338:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2785.880373] Lustre: server umount lustre-OST0000 complete [ 2785.882259] LustreError: 100338:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2796.493362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2796.766601] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2796.780770] LustreError: 98242:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2796.795458] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2796.802299] LustreError: 100805:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2796.807030] LustreError: 100805:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2796.811338] LustreError: 100805:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2796.878928] Lustre: server umount lustre-OST0000 complete [ 2796.880830] LustreError: 100805:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2808.284085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2808.509672] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2808.521812] Lustre: Found index 0 for lustre-OST0000, updating log [ 2808.527121] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2808.629312] Lustre: lustre-OST0000: new disk, initializing [ 2808.632981] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2808.640806] Lustre: 101339:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2808.878090] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2815.996378] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2820.116822] Lustre: Failing over lustre-OST0000 [ 2820.358502] Lustre: server umount lustre-OST0000 complete [ 2825.696816] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.201.106@tcp failed: rc = -107 [ 2825.708427] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2825.721083] Lustre: Skipped 2 previous similar messages [ 2825.728102] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 2829.281595] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 2829.300463] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2830.786583] Lustre: server umount lustre-MDT0000 complete [ 2834.409249] LustreError: 98248:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2834.437863] LustreError: 98248:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 2835.416471] LustreError: 98235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784940671 with bad export cookie 17097179101408859614 [ 2835.430266] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2835.439359] LustreError: 98235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2845.850446] Lustre: server umount lustre-MDT0001 complete [ 2855.591752] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 20:51:29 (1784940689) [ 2875.083202] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 2890.696651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2891.393545] LustreError: 103686:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2891.564751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2897.685897] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2907.641715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2914.892425] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2918.464245] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2926.118966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2927.007890] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2927.011707] Lustre: Skipped 1 previous similar message [ 2927.022735] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2933.164630] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2933.198992] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2933.200526] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.201.106@tcp (at 192.168.201.106@tcp) [ 2933.406839] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2942.456669] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2942.544673] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2942.653422] Lustre: lustre-OST0001: new disk, initializing [ 2942.660914] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2942.670412] Lustre: 106005:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2949.278245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2949.294621] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2949.372504] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2949.409585] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2957.487447] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2968.790790] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2975.213961] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 2975.239899] Lustre: Skipped 2 previous similar messages [ 2975.241466] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.106@tcp (stopping) [ 2975.260163] Lustre: Skipped 3 previous similar messages [ 2980.166901] Lustre: server umount lustre-MDT0001 complete [ 2980.331276] LustreError: 104418:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2980.349963] LustreError: 104418:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 5 previous similar messages [ 2991.075189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2991.143630] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2991.148508] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2991.220939] Lustre: 103676:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2992.209892] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2992.219626] Lustre: Skipped 1 previous similar message [ 2993.160502] LustreError: 107556:0:(obd_mount.c:230:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2993.176814] LustreError: 107556:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2993.200226] LustreError: 103678:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2993.337440] LustreError: 107567:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.106@tcp: cfg command failed: rc = -17 [ 2993.356522] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2993.356522] [ 2997.172654] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2997.236122] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 192.168.201.106@tcp (at 192.168.201.106@tcp) [ 3008.417926] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 20:54:02 (1784940842) [ 3027.160958] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 3048.010869] Lustre: 109622:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3072.380563] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3075.914731] Lustre: 110757:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3165.666961] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.201.106@tcp failed: rc = -107 [ 3165.682270] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3165.700246] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 3165.709401] Lustre: Skipped 1 previous similar message [ 3168.877957] Lustre: server umount lustre-MDT0000 complete [ 3171.306826] LustreError: 106745:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3171.323235] LustreError: 106745:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 11 previous similar messages [ 3172.715156] LustreError: 106004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784941009 with bad export cookie 17097179101408860615 [ 3172.721395] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3172.724808] LustreError: 106004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3173.055893] Lustre: server umount lustre-MDT0001 complete [ 3187.507600] Lustre: server umount lustre-OST0000 complete [ 3201.419811] Lustre: server umount lustre-OST0001 complete [ 3208.241429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3219.899855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3229.038652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3239.691203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3260.643077] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 3271.859500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3271.882111] Lustre: lustre-MDT0000: reset Object Index mappings [ 3272.615905] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3278.606190] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3288.540297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3288.562693] Lustre: lustre-MDT0001: reset Object Index mappings [ 3295.525836] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3299.390744] Lustre: 115858:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3308.121077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3314.214875] LustreError: 116211:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3314.234676] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:226 to 0x280000400:257) [ 3314.236072] LustreError: 116211:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 4 previous similar messages [ 3315.434438] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3317.316748] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 3325.630584] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3331.605859] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3331.618619] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3334.814369] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3344.594974] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3348.997642] Lustre: 117732:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3405.419462] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 21:00:39 (1784941239) [ 3432.420118] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.201.106@tcp failed: rc = -107 [ 3432.429787] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 3432.445189] Lustre: Skipped 3 previous similar messages [ 3432.454273] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 3432.462778] Lustre: Skipped 4 previous similar messages [ 3437.367937] Lustre: server umount lustre-MDT0000 complete [ 3441.714706] LustreError: 117734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784941278 with bad export cookie 17097179101408907445 [ 3441.726554] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3441.747493] LustreError: 117734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3448.576702] Lustre: server umount lustre-MDT0001 complete [ 3459.666916] Lustre: server umount lustre-OST0000 complete [ 3470.237295] Lustre: server umount lustre-OST0001 complete [ 3477.943870] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 3492.818994] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 3535.663370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3536.069807] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3536.118289] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3536.247885] Lustre: lustre-MDT0000: new disk, initializing [ 3536.374332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3536.381290] Lustre: Skipped 3 previous similar messages [ 3536.395878] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3541.300182] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3548.379953] Lustre: server umount lustre-MDT0000 complete [ 3558.371426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3558.735927] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3558.767894] Lustre: 122038:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3558.786269] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3564.005986] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3569.014040] Lustre: server umount lustre-MDT0000 complete [ 3579.042276] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 21:03:33 (1784941413) [ 3580.963726] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3582.949642] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 21:03:37 (1784941417) [ 3601.296419] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 3611.452912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3616.957190] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3625.788869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3625.918772] Lustre: 124089:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3625.967744] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3625.971557] Lustre: Skipped 1 previous similar message [ 3626.029954] Lustre: lustre-MDT0001: new disk, initializing [ 3626.167059] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3626.212796] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3630.738263] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3635.715281] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3642.935161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3643.119810] Lustre: lustre-OST0000: new disk, initializing [ 3643.124699] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3643.131744] Lustre: 125665:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3645.172973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3645.207690] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3645.382878] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3648.732485] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3658.657603] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3658.862770] Lustre: lustre-OST0001: new disk, initializing [ 3658.865539] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3658.871432] Lustre: 126537:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3660.688733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3660.703248] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3660.826305] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3665.368253] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3675.302439] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3679.187588] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4022.502556] Lustre: 132938:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4022.508572] Lustre: 132938:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4022.516262] Lustre: 132938:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4022.524293] Lustre: 132938:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4022.532330] Lustre: 132938:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4022.540918] Lustre: 132938:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4262.500600] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 21:14:57 (1784942097) [ 4276.705357] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.201.106@tcp failed: rc = -107 [ 4276.710397] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 4276.716426] Lustre: Skipped 5 previous similar messages [ 4276.720306] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 4276.726026] Lustre: Skipped 8 previous similar messages [ 4280.293269] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 4280.298067] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 4280.314401] Lustre: Skipped 2 previous similar messages [ 4282.077745] Lustre: server umount lustre-MDT0000 complete [ 4284.548162] LustreError: 124081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784942120 with bad export cookie 17097179101408976234 [ 4284.550872] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4284.556137] LustreError: 124081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4285.409180] LustreError: 127340:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4285.410850] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 4285.419911] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.106@tcp (stopping) [ 4285.419919] Lustre: Skipped 3 previous similar messages [ 4285.424436] LustreError: 127340:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 13 previous similar messages [ 4285.441043] Lustre: Skipped 1 previous similar message [ 4290.946755] Lustre: server umount lustre-MDT0001 complete [ 4299.506565] Lustre: server umount lustre-OST0000 complete [ 4316.127417] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4316.272711] Lustre: server umount lustre-OST0001 complete [ 4320.801767] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 4328.008885] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 4352.203836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4352.330190] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4352.346515] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4352.398896] Lustre: lustre-MDT0000: new disk, initializing [ 4352.449602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4352.453487] Lustre: Skipped 5 previous similar messages [ 4352.461361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4355.744988] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 4363.456609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4363.541987] Lustre: 135980:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4363.582451] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4363.585757] Lustre: Skipped 1 previous similar message [ 4363.642575] Lustre: lustre-MDT0001: new disk, initializing [ 4363.736251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4363.744134] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4366.917200] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 4373.567322] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4377.088404] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4380.989945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4381.136383] Lustre: lustre-OST0000: new disk, initializing [ 4381.139065] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4381.144284] Lustre: 137948:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4382.220034] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4382.227509] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4382.303774] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4384.655108] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 4390.780158] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4397.493661] Lustre: lustre-MDD0000: changelog on [ 4399.568521] Lustre: lustre-MDD0001: changelog on [ 5834.159816] Lustre: lustre-MDD0001: changelog off [ 5835.556213] Lustre: 135987:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5836.184731] Lustre: lustre-MDD0000: changelog off [ 5841.382842] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 5841.393739] Lustre: lustre-OST0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 5841.396781] Lustre: Skipped 4 previous similar messages [ 5845.646995] Lustre: server umount lustre-OST0000 complete [ 5848.545209] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.201.106@tcp failed: rc = -107 [ 5848.550840] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 5848.558469] Lustre: Skipped 1 previous similar message [ 5848.561828] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 5851.617812] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 5851.619157] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 5854.624608] Lustre: server umount lustre-MDT0000 complete [ 5856.743452] LustreError: 135991:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5856.760109] LustreError: 135991:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 3 previous similar messages [ 5857.006514] LustreError: 135971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784943693 with bad export cookie 17097179101409795535 [ 5857.013946] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5857.227372] Lustre: server umount lustre-MDT0001 complete [ 5863.908546] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 5865.573353] Key type lgssc unregistered [ 5865.746850] LNet: 147011:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5865.751867] LNetError: 147011:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5865.768598] LNet: Removed LNI 192.168.201.106@tcp [ 5866.324174] Key type .llcrypt unregistered [ 5866.327195] Key type ._llcrypt unregistered [ 5878.466794] Key type ._llcrypt registered [ 5878.468885] Key type .llcrypt registered [ 5878.526480] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 5887.567628] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 5888.204166] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5888.229584] alg: No test for adler32 (adler32-zlib) [ 5889.143266] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 5889.251038] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 5890.855299] Key type lgssc registered [ 5891.511578] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5909.726217] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5909.752073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5910.931714] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5910.949874] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5911.002537] Lustre: lustre-MDT0000: new disk, initializing [ 5911.048459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5911.063069] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5913.114628] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 5919.267241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5919.318104] Lustre: 150190:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5919.353754] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5919.356353] Lustre: Skipped 1 previous similar message [ 5919.393961] Lustre: lustre-MDT0001: new disk, initializing [ 5919.449715] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5919.471919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5919.483280] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5921.257541] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 5926.630156] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5929.694776] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5933.081447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5933.278686] Lustre: lustre-OST0000: Not available for connect from 192.168.201.106@tcp (not set up) [ 5933.281372] Lustre: lustre-OST0000: new disk, initializing [ 5933.289398] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5933.294050] Lustre: 152161:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5933.354468] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5936.368858] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 5938.696087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5938.702594] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5938.724854] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 5941.895186] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5946.341140] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_module obdecho/obdecho [ 5947.339381] LustreError: 153160:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 5947.343404] LustreError: 153160:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 5948.310625] LustreError: 153256:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 5955.558283] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.201.106@tcp failed: rc = -107 [ 5955.569202] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 5955.582863] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 5956.584829] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 5956.585204] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 5956.602639] Lustre: Skipped 1 previous similar message [ 5959.890217] Lustre: server umount lustre-MDT0000 complete [ 5962.025435] LustreError: 150184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784943798 with bad export cookie 13353356837329587172 [ 5962.028941] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5962.031307] LustreError: 150184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5962.255229] Lustre: server umount lustre-MDT0001 complete [ 5974.652764] Lustre: server umount lustre-OST0000 complete [ 5987.435090] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 5992.542793] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 6011.459462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6011.604746] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6011.637472] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6011.694649] Lustre: lustre-MDT0000: new disk, initializing [ 6011.758436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6011.770595] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6014.211697] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6020.396316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6020.448054] Lustre: 156971:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6020.481691] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6020.487685] Lustre: Skipped 1 previous similar message [ 6020.540901] Lustre: lustre-MDT0001: new disk, initializing [ 6020.591233] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6020.619995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6020.624131] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6022.827843] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6028.488117] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6031.607263] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6035.960852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6036.118104] Lustre: lustre-OST0000: new disk, initializing [ 6036.120806] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6036.124504] Lustre: 158936:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6036.193247] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6040.041937] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6045.232422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6045.242684] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6045.342870] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6046.689605] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6049.556228] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6049.705639] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6052.009293] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6052.112251] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6055.399038] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6055.407990] Lustre: lustre-OST0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 6055.412750] Lustre: Skipped 3 previous similar messages [ 6059.321491] Lustre: server umount lustre-OST0000 complete [ 6061.536555] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.201.106@tcp failed: rc = -107 [ 6061.539706] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6061.557459] Lustre: Skipped 1 previous similar message [ 6061.560153] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 6065.635415] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6067.697102] LustreError: 160474:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x2:0x0].0x0 (00000000d7f721d5) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 6069.754340] Lustre: server umount lustre-MDT0000 complete [ 6070.756456] LustreError: 158220:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6070.767754] LustreError: 158220:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 5 previous similar messages [ 6071.778537] LustreError: 156962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784943908 with bad export cookie 13353356837329588131 [ 6071.778978] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6071.783174] LustreError: 156962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6071.991456] Lustre: server umount lustre-MDT0001 complete [ 6074.076967] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 21:45:09 (1784943909) [ 6074.952304] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6076.022028] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 21:45:11 (1784943911) [ 6080.294102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6080.630398] LustreError: 161344:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6080.682260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6083.163810] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6086.112796] LustreError: 161345:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6088.040432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6090.629788] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6095.300723] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6098.694886] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6102.760071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6106.296290] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6110.273623] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6118.881431] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.201.106@tcp failed: rc = -107 [ 6118.886704] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6118.895754] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 6118.898826] Lustre: Skipped 2 previous similar messages [ 6122.227804] Lustre: server umount lustre-MDT0000 complete [ 6123.492597] LustreError: 163836:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6124.447526] LustreError: 161326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784943960 with bad export cookie 13353356837329589013 [ 6124.451087] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6124.453447] LustreError: 161326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6128.608113] LustreError: 161341:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6128.615478] LustreError: 161341:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 3 previous similar messages [ 6130.716554] Lustre: server umount lustre-MDT0001 complete [ 6134.568244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6142.364566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6142.527407] LustreError: 165183:0:(llog.c:1655:llog_backup()) MGC192.168.201.106@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6142.531762] Lustre: 165183:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.106@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6145.180265] LustreError: 165188:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6145.196976] LustreError: 165188:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 6162.785091] LustreError: 165189:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6162.796542] LustreError: 165189:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 10 previous similar messages [ 6197.596135] LustreError: 165190:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6197.606400] LustreError: 165190:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 25 previous similar messages [ 6203.871422] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6203.875607] Lustre: 165183:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.106@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6244.374750] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6244.377595] Lustre: Skipped 2 previous similar messages [ 6246.474121] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6249.444934] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 192.168.201.106@tcp (at 192.168.201.106@tcp) [ 6250.905919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6262.110686] LustreError: 165189:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6262.117116] LustreError: 165189:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 42 previous similar messages [ 6268.895793] LustreError: 148417:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000faa27f53 x1871649151497344/t0(0) o250->MGC192.168.201.106@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6271.329510] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6274.545542] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 192.168.201.106@tcp (at 192.168.201.106@tcp) [ 6275.349508] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6276.237793] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6278.942651] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6279.925658] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6290.505624] Lustre: server umount lustre-MDT0000 complete [ 6292.421882] LustreError: 161326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784944128 with bad export cookie 13353356837329595460 [ 6292.425379] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6292.427653] LustreError: 161326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6295.009563] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6295.010424] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.106@tcp (stopping) [ 6295.015194] Lustre: Skipped 4 previous similar messages [ 6295.018073] Lustre: Skipped 4 previous similar messages [ 6298.651331] Lustre: server umount lustre-MDT0001 complete [ 6300.534775] Lustre: server umount lustre-OST0000 complete [ 6310.910543] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 6316.051884] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 6333.115836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6333.239341] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6333.260782] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6333.304138] Lustre: lustre-MDT0000: new disk, initializing [ 6333.360149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6333.363328] Lustre: Skipped 1 previous similar message [ 6333.380297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6335.348824] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6340.533089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6340.605350] Lustre: 170259:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6340.774759] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6342.792282] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6347.729518] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6350.404636] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6353.922268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6354.080456] Lustre: 172226:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6357.028519] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6361.821442] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6363.577606] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6364.182247] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6364.246786] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6364.724857] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 6366.819298] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6366.929530] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6369.252472] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.201.106@tcp failed: rc = -107 [ 6369.253332] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6369.263354] Lustre: lustre-OST0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 6369.267956] Lustre: Skipped 2 previous similar messages [ 6369.885444] Lustre: server umount lustre-OST0000 complete [ 6377.711597] Lustre: server umount lustre-MDT0000 complete [ 6379.751262] LustreError: 170250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784944216 with bad export cookie 13353356837329612281 [ 6379.754325] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6379.756984] LustreError: 170250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6379.922409] Lustre: server umount lustre-MDT0001 complete [ 6384.729347] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 21:50:19 (1784944219) [ 6388.235857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6390.550868] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6393.827712] LustreError: 174640:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6393.835216] LustreError: 174640:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 12 previous similar messages [ 6395.067525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6397.701539] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6401.426527] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6404.438368] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6408.164516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6411.393097] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6415.554828] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6425.063254] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 21:51:00 (1784944260) [ 6429.152803] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.201.106@tcp failed: rc = -107 [ 6429.155852] LustreError: Skipped 2 previous similar messages [ 6432.802198] Lustre: server umount lustre-OST0000 complete [ 6436.322894] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6436.331026] Lustre: Skipped 5 previous similar messages [ 6436.334168] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 6436.336275] Lustre: Skipped 7 previous similar messages [ 6440.421819] Lustre: server umount lustre-MDT0000 complete [ 6441.957348] LustreError: 174621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784944278 with bad export cookie 13353356837329613149 [ 6441.960261] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6441.965127] LustreError: 174621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6442.095446] Lustre: server umount lustre-MDT0001 complete [ 6446.398824] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 6447.724938] Key type lgssc unregistered [ 6447.859829] LNet: 178362:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6447.866275] LNetError: 178362:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6447.873469] LNet: Removed LNI 192.168.201.106@tcp [ 6448.278327] Key type .llcrypt unregistered [ 6448.280088] Key type ._llcrypt unregistered [ 6457.097298] Key type ._llcrypt registered [ 6457.098714] Key type .llcrypt registered [ 6457.151727] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 6457.511507] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6457.550067] alg: No test for adler32 (adler32-zlib) [ 6458.427525] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 6458.550703] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 6460.151241] Key type lgssc registered [ 6460.652852] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6463.969970] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6463.979879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6479.455376] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6484.575375] LustreError: 179581:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.106@tcp: failed processing log, type 4: rc = -110 [ 6489.696256] LustreError: 179581:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6489.702185] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 6489.704743] LustreError: MGC192.168.201.106@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 6489.709777] LustreError: 179581:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 6489.713773] LustreError: 179581:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 6489.716410] LustreError: 179581:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 6489.740386] Lustre: server umount lustre-OST0000 complete [ 6489.742382] LustreError: 179581:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6493.254749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6494.449471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6496.201179] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6499.882962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6500.084308] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6501.700735] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6504.596489] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6506.845075] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6509.708751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6509.805306] LustreError: 181891:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6509.810332] Lustre: 181891:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.201.106@tcp: invalid local config log lustre-OST0000: rc = -22 [ 6509.878279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6512.628564] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6516.487461] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6520.802184] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.201.106@tcp failed: rc = -107 [ 6520.809234] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6520.823445] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 6521.322162] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6521.324720] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 6521.336990] Lustre: Skipped 1 previous similar message [ 6521.344506] Lustre: Skipped 1 previous similar message [ 6523.862425] Lustre: server umount lustre-MDT0000 complete [ 6525.735871] LustreError: 180059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784944362 with bad export cookie 7080832654352269627 [ 6525.738187] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6525.745161] LustreError: 180059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6525.918982] Lustre: server umount lustre-MDT0001 complete [ 6538.437308] Lustre: server umount lustre-OST0000 complete [ 6543.935864] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 6548.884623] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 6565.280832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6565.394565] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6565.406318] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6565.445456] Lustre: lustre-MDT0000: new disk, initializing [ 6565.485212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6565.495913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6567.509964] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6573.328620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6573.374422] Lustre: 185412:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6573.405588] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6573.407867] Lustre: Skipped 1 previous similar message [ 6573.445231] Lustre: lustre-MDT0001: new disk, initializing [ 6573.491570] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6573.508763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6573.512943] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6575.537790] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6580.758861] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6583.606658] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6586.812588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6586.975171] Lustre: lustre-OST0000: new disk, initializing [ 6586.977473] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6586.981314] Lustre: 187380:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6587.022633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6589.786534] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6593.544192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6593.549692] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6593.591434] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6594.794499] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6596.662393] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6596.766205] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6598.469781] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6598.603064] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6603.753499] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6603.755225] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.201.106@tcp failed: rc = -107 [ 6603.755511] Lustre: lustre-OST0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 6603.755523] Lustre: Skipped 1 previous similar message [ 6603.757664] Lustre: Skipped 1 previous similar message [ 6603.762408] LustreError: Skipped 1 previous similar message [ 6605.615433] Lustre: server umount lustre-OST0000 complete [ 6608.865099] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6608.866903] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 6608.870950] Lustre: Skipped 1 previous similar message [ 6608.873456] Lustre: Skipped 1 previous similar message [ 6613.734165] Lustre: server umount lustre-MDT0000 complete [ 6613.984527] LustreError: 185424:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6613.992721] LustreError: 185424:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 3 previous similar messages [ 6615.547578] LustreError: 185407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784944451 with bad export cookie 7080832654352270327 [ 6615.551422] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6615.552923] LustreError: 185407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6615.750527] Lustre: server umount lustre-MDT0001 complete [ 6620.152819] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 21:54:15 (1784944455) [ 6628.120204] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 6629.481849] Key type lgssc unregistered [ 6629.633674] LNet: 190320:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6629.636746] LNetError: 190320:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6629.644337] LNet: Removed LNI 192.168.201.106@tcp [ 6630.002419] Key type .llcrypt unregistered [ 6630.003910] Key type ._llcrypt unregistered [ 6638.617971] Key type ._llcrypt registered [ 6638.619302] Key type .llcrypt registered [ 6638.672902] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 6639.017768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6639.026642] alg: No test for adler32 (adler32-zlib) [ 6639.911805] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 6640.017947] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 6641.623186] Key type lgssc registered [ 6642.067071] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6645.307180] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6645.313567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6646.513421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6648.259594] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6652.063631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6652.258902] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6654.375630] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6658.031909] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6660.810617] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6664.490469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6664.608574] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6664.610766] LustreError: 193380:0:(llog.c:1696:llog_backup()) MGC192.168.201.106@tcp: failed to backup log lustre-OST0000: rc = -28 [ 6664.614948] Lustre: 193380:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.201.106@tcp: can't backup local config lustre-OST0000: rc = -28 [ 6664.668446] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6667.485728] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6670.889130] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6672.864503] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.201.106@tcp failed: rc = -107 [ 6672.867509] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6672.873427] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 6675.936895] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 6675.938336] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6676.961495] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 6676.963424] Lustre: Skipped 4 previous similar messages [ 6682.087046] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 6682.091196] Lustre: Skipped 2 previous similar messages [ 6686.175110] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6686.270729] Lustre: server umount lustre-MDT0000 complete [ 6687.202397] LustreError: 192669:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6687.210705] LustreError: 192669:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 4 previous similar messages [ 6687.896971] LustreError: 191496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784944524 with bad export cookie 3545572658189582038 [ 6687.899728] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6687.904689] LustreError: 191496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6689.247426] LustreError: 192244:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000008d67ca4b x1871649938890752/t0(0) o900->lustre-MDT0000-lwp-MDT0001@192.168.201.106@tcp:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6689.352573] Lustre: server umount lustre-MDT0001 complete [ 6701.310506] Lustre: server umount lustre-OST0000 complete [ 6705.723422] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 6711.533681] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 6731.610196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6731.755268] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6731.771475] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6731.817520] Lustre: lustre-MDT0000: new disk, initializing [ 6731.857845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6731.869879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6734.439804] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6740.418821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6740.498310] Lustre: 196902:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6740.539750] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6740.541262] Lustre: Skipped 1 previous similar message [ 6740.543519] LustreError: 197681:0:(llog.c:1696:llog_backup()) MGC192.168.201.106@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 6740.547522] LustreError: 197681:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 6740.550849] Lustre: 197681:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.106@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 6740.562169] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6740.564531] Lustre: Skipped 1 previous similar message [ 6740.607814] Lustre: lustre-MDT0001: new disk, initializing [ 6740.653117] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6740.674450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6740.681836] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6742.858895] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6748.251201] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6750.910934] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6753.970839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6754.083065] Lustre: *** cfs_fail_loc=131e, val=0*** [ 6754.084548] Lustre: Skipped 1 previous similar message [ 6754.086431] LustreError: 198843:0:(llog.c:1696:llog_backup()) MGC192.168.201.106@tcp: failed to backup log lustre-OST0000: rc = -28 [ 6754.090330] LustreError: 198843:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 6754.092921] Lustre: 198843:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.201.106@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 6754.096453] Lustre: 198843:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 6754.107235] Lustre: lustre-OST0000: new disk, initializing [ 6754.109558] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6754.112391] Lustre: 198868:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6754.144309] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6756.375782] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6760.949086] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6760.971960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6760.976411] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6761.025790] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6762.738743] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6762.848734] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6764.533207] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6764.650357] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6766.052649] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.201.106@tcp failed: rc = -107 [ 6766.052767] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6766.053865] Lustre: lustre-OST0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 6766.063510] Lustre: Skipped 2 previous similar messages [ 6767.651392] Lustre: server umount lustre-OST0000 complete [ 6771.169421] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6771.173874] Lustre: Skipped 1 previous similar message [ 6775.340769] Lustre: server umount lustre-MDT0000 complete [ 6776.291483] LustreError: 196912:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6776.306562] LustreError: 196912:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 1 previous similar message [ 6777.367430] LustreError: 196894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784944613 with bad export cookie 3545572658189582759 [ 6777.369208] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6777.373523] LustreError: 196894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6777.551861] Lustre: server umount lustre-MDT0001 complete [ 6781.327259] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 21:56:56 (1784944616) [ 6788.051393] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 6791.890454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6792.053526] LustreError: 201989:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6792.073524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6793.663358] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6797.282105] LustreError: 201990:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6797.450174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6799.958392] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6801.436658] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 6804.556040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6807.601173] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6811.309082] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6811.384949] Lustre: lustre-OST0001: new disk, initializing [ 6811.387226] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6811.390450] Lustre: 204293:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6814.167982] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6816.288444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6816.292886] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6816.347754] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6820.689193] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 6823.983971] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 6825.044588] Lustre: DEBUG MARKER: ADD OST3 [ 6827.306044] loop: module loaded [ 6827.331261] loop0: detected capacity change from 0 to 204800000 [ 6827.441905] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6827.450740] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6827.454762] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6830.296411] loop0: detected capacity change from 0 to 204800000 [ 6832.758026] Lustre: DEBUG MARKER: STOP OST3 [ 6833.030187] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6833.130176] Lustre: lustre-OST0003: new disk, initializing [ 6833.136981] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 6833.144776] Lustre: 206098:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6833.245442] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 6833.248227] Lustre: Skipped 3 previous similar messages [ 6834.478024] Lustre: *** cfs_fail_loc=2109, val=2*** [ 6834.554372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 6834.561691] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 6834.583102] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 6835.624079] Lustre: Failing over lustre-OST0003 [ 6835.708598] Lustre: server umount lustre-OST0003 complete [ 6836.473961] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6836.511131] LustreError: 206366:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 6839.666451] LustreError: 203472:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.201.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6839.786623] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6839.789738] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 192.168.201.106@tcp failed: rc = -107 [ 6839.792101] Lustre: Skipped 1 previous similar message [ 6839.795114] LustreError: Skipped 1 previous similar message [ 6841.793951] Lustre: DEBUG MARKER: START OST3 again [ 6845.108077] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6845.243294] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 6846.564357] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 6846.965730] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 6846.969321] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 192.168.201.106@tcp (at 192.168.201.106@tcp) [ 6848.064057] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6862.451938] Lustre: Failing over lustre-OST0003 [ 6862.494092] Lustre: server umount lustre-OST0003 complete [ 6865.896056] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6865.901734] LustreError: 208261:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6865.908750] LustreError: 208261:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 6 previous similar messages [ 6876.139881] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 6876.142806] Lustre: Skipped 6 previous similar messages [ 6881.388778] Lustre: server umount lustre-MDT0000 complete [ 6882.840514] LustreError: 201971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784944719 with bad export cookie 3545572658189583641 [ 6882.845291] LustreError: 201971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6882.845375] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6886.368809] LustreError: 201984:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6886.369686] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6886.375025] LustreError: 201984:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 5 previous similar messages [ 6886.379942] Lustre: Skipped 5 previous similar messages [ 6889.097750] Lustre: server umount lustre-MDT0001 complete [ 6890.897930] Lustre: server umount lustre-OST0000 complete [ 6892.544301] Lustre: server umount lustre-OST0001 complete [ 6895.346794] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 6898.761892] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 6914.395185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6914.482121] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6914.493634] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6914.531636] Lustre: lustre-MDT0000: new disk, initializing [ 6914.555382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6914.557677] Lustre: Skipped 1 previous similar message [ 6914.562939] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6916.355518] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6921.101225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6921.145290] Lustre: 211595:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6921.262463] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6922.865558] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6926.962820] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6929.154940] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6931.863363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6931.967377] Lustre: lustre-OST0000: new disk, initializing [ 6931.969303] Lustre: Skipped 1 previous similar message [ 6931.971319] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6931.974370] Lustre: Skipped 2 previous similar messages [ 6931.977379] Lustre: 213557:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6933.064497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6933.067722] Lustre: Skipped 1 previous similar message [ 6933.075609] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6933.131646] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6934.535274] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6938.967196] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6940.585076] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6940.667483] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6942.245513] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6942.328761] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6943.715157] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.201.106@tcp failed: rc = -107 [ 6943.715730] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 6943.716917] Lustre: lustre-OST0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 6943.716922] Lustre: Skipped 8 previous similar messages [ 6943.718362] LustreError: Skipped 1 previous similar message [ 6943.728732] Lustre: Skipped 2 previous similar messages [ 6949.207990] Lustre: server umount lustre-OST0000 complete [ 6951.907295] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.201.106@tcp failed: rc = -107 [ 6965.215170] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6965.298161] Lustre: server umount lustre-MDT0000 complete [ 6966.917382] LustreError: 212387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784944803 with bad export cookie 3545572658189585923 [ 6966.920502] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6966.923304] LustreError: 212387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6967.073467] Lustre: server umount lustre-MDT0001 complete [ 6968.479144] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 22:00:03 (1784944803) [ 6974.657806] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 6977.980754] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 6991.659146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6991.768111] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6991.783893] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6991.820431] Lustre: lustre-MDT0000: new disk, initializing [ 6991.862157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6993.677929] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6999.154160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6999.218845] Lustre: 218166:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6999.349723] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7001.132618] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7005.632420] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7007.564076] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7010.043892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7010.155914] Lustre: 220134:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7011.646983] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7012.624387] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7016.796199] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7018.436277] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7018.536742] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7020.287857] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7020.375534] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7022.051079] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.201.106@tcp failed: rc = -107 [ 7022.052358] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 7022.055015] LustreError: Skipped 1 previous similar message [ 7022.066715] Lustre: Skipped 2 previous similar messages [ 7022.069606] Lustre: lustre-OST0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 7022.075715] Lustre: Skipped 10 previous similar messages [ 7027.300910] Lustre: server umount lustre-OST0000 complete [ 7035.113128] Lustre: server umount lustre-MDT0000 complete [ 7036.804256] LustreError: 220133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784944873 with bad export cookie 3545572658189586805 [ 7036.807605] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7036.808051] LustreError: 220133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7036.941785] Lustre: server umount lustre-MDT0001 complete [ 7040.130075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7040.319083] LustreError: 222357:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7040.328043] LustreError: 222357:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 7041.966363] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7045.575097] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7045.733939] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7045.736552] Lustre: Skipped 6 previous similar messages [ 7047.283038] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7050.046993] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7052.061336] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7054.495216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7056.862335] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7059.515148] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7091.353277] Lustre: DEBUG MARKER: conf-sanity test_153a: @@@@@@ FAIL: check_mount failed [ 7113.735820] Lustre: DEBUG MARKER: == conf-sanity test 153b: added IPv6 NID support ========= 22:02:28 (1784944948) [ 7126.500808] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 7126.503541] Lustre: Skipped 7 previous similar messages [ 7131.554213] Lustre: server umount lustre-MDT0000 complete [ 7131.617559] LustreError: 222354:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7131.624023] LustreError: 222354:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 3 previous similar messages [ 7132.924489] LustreError: 222338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784944969 with bad export cookie 3545572658189587687 [ 7132.928289] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7132.928847] LustreError: 222338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7139.121571] Lustre: server umount lustre-MDT0001 complete [ 7146.665230] Lustre: server umount lustre-OST0000 complete [ 7150.268737] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 7154.224858] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 7168.425235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7168.541314] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7168.559453] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7168.561575] Lustre: Skipped 3 previous similar messages [ 7168.599987] Lustre: lustre-MDT0000: new disk, initializing [ 7168.601772] Lustre: Skipped 2 previous similar messages [ 7168.635927] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7168.639433] Lustre: Skipped 2 previous similar messages [ 7170.170097] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7174.947192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7174.994494] Lustre: 228109:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7175.119699] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7175.125450] Lustre: Skipped 1 previous similar message [ 7176.587945] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7180.336313] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7182.206864] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7184.670201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7184.800635] Lustre: 230077:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7186.511174] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7186.983832] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7190.915355] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7192.471497] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7192.558940] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7194.124752] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7194.208786] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7196.646064] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.201.106@tcp failed: rc = -107 [ 7196.646879] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 7196.649130] LustreError: Skipped 1 previous similar message [ 7196.654933] Lustre: Skipped 6 previous similar messages [ 7201.048416] Lustre: server umount lustre-OST0000 complete [ 7208.543411] Lustre: server umount lustre-MDT0000 complete [ 7210.001548] LustreError: 228100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784945046 with bad export cookie 3545572658189588478 [ 7210.006130] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7210.009755] LustreError: 228100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7210.142450] Lustre: server umount lustre-MDT0001 complete [ 7217.249267] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 7221.375804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7223.122775] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7226.547756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7228.219390] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7229.452085] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7232.575821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7235.071879] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7238.655777] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7238.734499] Lustre: 235311:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7240.391942] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7240.394937] Lustre: Skipped 1 previous similar message [ 7240.442793] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7241.237344] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7245.856947] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7436.194067] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 22:07:51 (1784945271) [ 7439.069645] Lustre: server umount lustre-MDT0000 complete [ 7440.549320] LustreError: 232988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784945276 with bad export cookie 3545572658189589360 [ 7440.554539] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7440.686459] Lustre: server umount lustre-MDT0001 complete [ 7456.735178] Lustre: 190828:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784945277/real 1784945277] req@ffff9936c3d52680 x1871649939302272/t0(0) o400->lustre-MDT0001-lwp-OST0001@192.168.201.106@tcp:12/10 lens 224/224 e 0 to 1 dl 1784945293 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7456.743530] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 7456.747620] Lustre: Skipped 3 previous similar messages [ 7460.831128] Lustre: 190828:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1784945281/real 1784945281] req@ffff9936cd492300 x1871649939302784/t0(0) o400->lustre-MDT0001-lwp-OST0001@192.168.201.106@tcp:12/10 lens 224/224 e 0 to 1 dl 1784945297 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 7460.845207] Lustre: 190828:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 7461.895982] Lustre: server umount lustre-OST0000 complete [ 7463.236488] Lustre: server umount lustre-OST0001 complete [ 7465.369607] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 7468.498030] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 7480.196534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7480.280065] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7480.283495] Lustre: Skipped 1 previous similar message [ 7480.291068] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7480.292963] Lustre: Skipped 4 previous similar messages [ 7480.328343] Lustre: lustre-MDT0000: new disk, initializing [ 7480.330298] Lustre: Skipped 3 previous similar messages [ 7480.353822] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7480.355555] Lustre: Skipped 8 previous similar messages [ 7480.360272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7480.362660] Lustre: Skipped 3 previous similar messages [ 7481.757901] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7485.898179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7485.937243] Lustre: 239233:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7486.046982] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7487.363763] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7491.200502] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7493.106109] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7495.323176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7495.443737] Lustre: 241202:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7497.191158] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7497.424310] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7501.059170] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7502.440871] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7502.516201] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7503.933466] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7504.002966] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7507.426804] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.201.106@tcp failed: rc = -107 [ 7507.428163] Lustre: lustre-OST0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 7507.429428] LustreError: Skipped 3 previous similar messages [ 7507.432732] Lustre: Skipped 8 previous similar messages [ 7510.739526] Lustre: server umount lustre-OST0000 complete [ 7526.367120] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7526.426415] Lustre: server umount lustre-MDT0000 complete [ 7526.880813] LustreError: 239245:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7526.886870] LustreError: 239245:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 4 previous similar messages [ 7527.819755] LustreError: 239227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784945364 with bad export cookie 3545572658189591348 [ 7527.825462] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7527.828379] LustreError: 239227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7527.943837] Lustre: server umount lustre-MDT0001 complete [ 7531.039562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7532.956885] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7536.593389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7538.302200] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7541.199352] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7543.242093] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7545.699403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7548.006642] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7550.498647] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7581.471871] Lustre: DEBUG MARKER: conf-sanity test_153c: @@@@@@ FAIL: too few connection attempts [ 7809.165275] Lustre: DEBUG MARKER: == conf-sanity test 154: expand .. on rename after MDT backup restore ========================================================== 22:14:04 (1784945644) [ 7812.067803] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 7812.069749] Lustre: Skipped 8 previous similar messages [ 7824.351132] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7824.417150] Lustre: server umount lustre-MDT0000 complete [ 7825.710707] LustreError: 243400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784945662 with bad export cookie 3545572658189592230 [ 7825.714267] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7825.714367] LustreError: 243400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7825.842054] Lustre: server umount lustre-MDT0001 complete [ 7837.340133] Lustre: server umount lustre-OST0000 complete [ 7840.387390] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 7843.726773] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 7858.134780] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 7861.170425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7861.243826] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7861.251869] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7861.253218] Lustre: Skipped 3 previous similar messages [ 7861.288000] Lustre: lustre-MDT0000: new disk, initializing [ 7861.289121] Lustre: Skipped 2 previous similar messages [ 7861.311554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7861.313593] Lustre: Skipped 2 previous similar messages [ 7862.515378] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7866.288554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7866.324238] Lustre: 249798:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7866.411796] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7866.413847] Lustre: Skipped 1 previous similar message [ 7867.609846] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7871.974374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7872.089699] Lustre: 251428:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7872.590766] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7874.040446] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7878.255622] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7880.236492] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7880.279406] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7884.630346] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7894.883310] Lustre: server umount lustre-MDT0000 complete [ 7895.523281] LustreError: 249808:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7895.527822] LustreError: 249808:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 5 previous similar messages [ 7896.256067] Lustre: server umount lustre-MDT0001 complete [ 7906.821949] Lustre: server umount lustre-OST0000 complete [ 7918.308871] Lustre: server umount lustre-OST0001 complete [ 7922.998137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7925.954142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7928.566152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7931.444198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7937.692382] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 7940.963873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7940.970234] Lustre: lustre-MDT0000: reset Object Index mappings [ 7942.383194] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7945.172941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7945.180730] Lustre: lustre-MDT0001: reset Object Index mappings [ 7946.690192] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7947.640053] Lustre: 258377:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 7949.798075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7950.952344] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7951.961620] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7954.982949] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7957.077193] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7960.230291] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7960.557157] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 7965.668836] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.201.106@tcp failed: rc = -107 [ 7968.668551] Lustre: server umount lustre-MDT0000 complete [ 7970.064861] Lustre: server umount lustre-MDT0001 complete [ 7981.567621] Lustre: server umount lustre-OST0000 complete [ 7993.113244] Lustre: server umount lustre-OST0001 complete [ 7999.364553] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 22:17:14 (1784945834) [ 8004.429026] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 8007.350304] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 8018.191152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8018.266900] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8018.269903] Lustre: Skipped 2 previous similar messages [ 8018.326442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8018.329038] Lustre: Skipped 13 previous similar messages [ 8019.610686] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8023.470787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8023.512798] Lustre: 264488:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8023.517963] Lustre: 264488:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 8024.797967] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8028.214339] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8029.812235] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8032.074651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8032.185252] Lustre: 266452:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8032.187888] Lustre: 266452:0:(osd_compat.c:1352:osd_object_spec_find()) Skipped 1 previous similar message [ 8033.832784] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8034.094630] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8037.841487] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8039.313515] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8039.386948] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8040.831450] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8040.901550] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8044.003180] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8044.007876] Lustre: Skipped 17 previous similar messages [ 8047.632694] Lustre: server umount lustre-OST0000 complete [ 8055.006644] Lustre: server umount lustre-MDT0000 complete [ 8056.430433] Lustre: server umount lustre-MDT0001 complete [ 8061.715677] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 8064.790086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8066.150591] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8068.706691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8070.146739] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8073.293198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8075.269956] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8078.125165] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8080.041755] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8080.288324] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8084.281203] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8086.984892] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 8086.988908] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 8094.871474] Lustre: server umount lustre-MDT0000 complete [ 8096.041225] LustreError: 270517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784945932 with bad export cookie 3545572658189602548 [ 8096.043428] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8096.046456] LustreError: 270517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8096.050080] LustreError: Skipped 3 previous similar messages [ 8096.159803] Lustre: server umount lustre-MDT0001 complete [ 8107.013311] Lustre: server umount lustre-OST0000 complete [ 8118.496958] Lustre: server umount lustre-OST0001 complete [ 8123.646282] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 8126.964988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8128.429689] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8131.093349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8132.487616] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8133.389606] Lustre: 275880:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 8135.415817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8137.392842] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8137.582084] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 8140.007966] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8141.107191] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 8141.853322] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8144.800258] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8147.385830] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 8147.394270] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 8150.224748] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 22:19:45 (1784945985) [ 8167.708645] Lustre: server umount lustre-MDT0000 complete [ 8183.263108] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8183.350915] Lustre: server umount lustre-MDT0001 complete [ 8190.716305] Lustre: server umount lustre-OST0000 complete [ 8206.303094] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8206.359966] Lustre: server umount lustre-OST0001 complete [ 8208.338654] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 8211.129512] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 8225.275200] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 8228.469917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8229.816488] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8233.758254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8235.127431] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8239.366562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8240.353462] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8241.288869] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8245.194408] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8246.816812] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8247.094118] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8251.356269] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8256.246035] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 22:21:31 (1784946091) [ 8263.000717] Lustre: server umount lustre-MDT0000 complete [ 8264.244504] Lustre: server umount lustre-MDT0001 complete [ 8275.459740] Lustre: server umount lustre-OST0000 complete [ 8285.851357] Lustre: server umount lustre-OST0001 complete [ 8288.724070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8290.398638] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8293.290028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8294.790383] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8297.253635] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8299.060114] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8301.358598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8303.371587] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8305.773558] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8322.221397] loop1: detected capacity change from 0 to 204800000 [ 8322.303475] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8322.308431] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8322.313020] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8323.998995] loop1: detected capacity change from 0 to 204800000 [ 8325.918292] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8325.950176] LustreError: 286716:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs27117: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 8325.954141] LustreError: 286716:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs27117 [ 8325.955833] LustreError: 286716:0:(mgs_handler.c:517:mgs_target_reg()) fs27117-MDT0001: Error checking target index: rc = -13 [ 8325.958816] LustreError: fs27117-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 8325.963157] LustreError: 290465:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 8325.966290] LustreError: 290465:0:(tgt_mount.c:2129:server_put_super()) no obd fs27117-MDT0001 [ 8325.969023] LustreError: 290465:0:(tgt_mount.c:116:server_deregister_mount()) fs27117-MDT0001 not registered [ 8325.981280] Lustre: server umount fs27117-MDT0001 complete [ 8325.982661] LustreError: 290465:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 8329.003815] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8329.039673] Lustre: Setting parameter fs27117-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs27117-MDT0001 [ 8329.044054] Lustre: Skipped 5 previous similar messages [ 8329.145260] Lustre: srv-fs27117-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 8329.147741] Lustre: 290930:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs27117-MDT0001-mdtlov: can't create new llog: rc = -115 [ 8329.150491] LustreError: 290930:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs27117-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 8330.614932] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8333.591418] Lustre: server umount fs27117-MDT0001 complete [ 8338.141587] loop1: detected capacity change from 0 to 204800000 [ 8340.054477] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8340.167089] Lustre: 292495:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs27117-MDT0001-mdtlov: can't create new llog: rc = -115 [ 8340.169592] LustreError: 292495:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs27117-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 8341.487961] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8343.075955] Lustre: server umount fs27117-MDT0001 complete [ 8346.978036] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.6@tcp (stopping) [ 8346.981338] Lustre: Skipped 47 previous similar messages [ 8349.668397] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.201.106@tcp failed: rc = -107 [ 8349.671078] LustreError: Skipped 4 previous similar messages [ 8352.475517] Lustre: server umount lustre-MDT0000 complete [ 8355.278868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8356.828509] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8359.403278] loop1: detected capacity change from 0 to 204800000 [ 8360.934048] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.201.106@tcp (at 192.168.201.106@tcp) [ 8360.936914] Lustre: Skipped 1 previous similar message [ 8361.295047] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8361.416819] Lustre: 294936:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs27117-MDT0001-mdtlov: can't create new llog: rc = -115 [ 8361.419384] LustreError: 294936:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs27117-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 8362.769014] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8364.433717] Lustre: server umount fs27117-MDT0001 complete [ 8376.973958] Lustre: server umount lustre-OST0000 complete [ 8384.343540] Lustre: server umount lustre-MDT0000 complete [ 8385.692609] Lustre: server umount lustre-MDT0001 complete [ 8389.388638] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 8390.448950] Key type lgssc unregistered [ 8390.575351] LNet: 297276:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8390.577652] LNetError: 297276:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8390.586328] LNet: Removed LNI 192.168.201.106@tcp [ 8390.924281] Key type .llcrypt unregistered [ 8390.925501] Key type ._llcrypt unregistered [ 8398.876976] Key type ._llcrypt registered [ 8398.878022] Key type .llcrypt registered [ 8398.924494] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 8399.365827] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8399.439273] alg: No test for adler32 (adler32-zlib) [ 8400.294828] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 8400.385857] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 8401.967103] Key type lgssc registered [ 8402.331302] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8404.917481] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8404.924867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8406.067209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8407.499198] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8410.321274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8410.465240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8411.860486] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8414.307235] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8416.198146] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8418.491907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8418.612973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8420.528708] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8420.646102] LustreError: 300350:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8420.655027] LustreError: 300350:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 8422.940772] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8425.955306] LustreError: 300735:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8431.077176] LustreError: 300350:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8433.507030] LustreError: 300351:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8433.512609] LustreError: 300351:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 8438.621303] LustreError: 300735:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8438.625873] LustreError: 300735:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 8439.536393] loop1: detected capacity change from 0 to 204800000 [ 8439.601261] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8439.604305] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8439.608399] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8441.386557] loop1: detected capacity change from 0 to 204800000 [ 8443.267248] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8443.321095] Lustre: fs3574-OST0000: new disk, initializing [ 8443.322662] Lustre: srv-fs3574-OST0000: No data found on store. Initialize space. [ 8443.324836] Lustre: 302278:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8443.348367] Lustre: fs3574-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8445.285372] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8447.460401] LustreError: 300737:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8447.464813] LustreError: 300737:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 4 previous similar messages [ 8448.650886] Lustre: server umount fs3574-OST0000 complete [ 8452.759393] loop1: detected capacity change from 0 to 204800000 [ 8454.598425] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8454.656542] Lustre: fs3574-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8456.520682] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8459.367092] loop2: detected capacity change from 0 to 204800000 [ 8459.447252] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8459.452633] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8459.458579] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8461.185115] loop2: detected capacity change from 0 to 204800000 [ 8463.137815] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8463.167345] LustreError: fs3574-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 8463.170911] LustreError: fs3574-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 8463.173487] LustreError: 305315:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 8463.175571] LustreError: 305315:0:(tgt_mount.c:2129:server_put_super()) no obd fs3574-OST0001 [ 8463.177244] LustreError: 305315:0:(tgt_mount.c:116:server_deregister_mount()) fs3574-OST0001 not registered [ 8463.187199] Lustre: server umount fs3574-OST0001 complete [ 8463.188400] LustreError: 305315:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 8464.220096] LustreError: 300350:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8464.224498] LustreError: 300350:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 9 previous similar messages [ 8466.529570] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8466.571871] Lustre: fs3574-OST0001: new disk, initializing [ 8466.573341] Lustre: srv-fs3574-OST0001: No data found on store. Initialize space. [ 8466.575695] Lustre: 305864:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8466.597423] Lustre: fs3574-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8468.527350] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8471.997255] Lustre: server umount fs3574-OST0000 complete [ 8475.075338] Lustre: server umount fs3574-OST0001 complete [ 8479.121166] loop1: detected capacity change from 0 to 204800000 [ 8480.949295] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8482.924727] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8485.162806] loop2: detected capacity change from 0 to 204800000 [ 8487.198305] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8487.267870] Lustre: fs3574-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8487.269950] Lustre: Skipped 1 previous similar message [ 8489.248099] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8492.168199] Lustre: server umount fs3574-OST0000 complete [ 8495.209382] Lustre: server umount fs3574-OST0001 complete [ 8498.659708] LustreError: 300735:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8498.665398] LustreError: 300735:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 19 previous similar messages [ 8502.755242] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.201.106@tcp failed: rc = -107 [ 8502.758957] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8502.767302] Lustre: lustre-OST0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 8503.782803] Lustre: lustre-OST0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 8503.783688] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8503.789178] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.201.106@tcp failed: rc = -107 [ 8505.065592] Lustre: server umount lustre-OST0000 complete [ 8507.872395] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.201.106@tcp failed: rc = -107 [ 8507.875572] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8507.881094] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 8507.883620] Lustre: Skipped 1 previous similar message [ 8512.676492] Lustre: server umount lustre-MDT0000 complete [ 8513.996864] LustreError: 298505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784946350 with bad export cookie 1046412681531954071 [ 8514.000126] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8514.002286] LustreError: 298505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8514.133345] Lustre: server umount lustre-MDT0001 complete [ 8517.949761] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 8518.952972] Key type lgssc unregistered [ 8519.072323] LNet: 311775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8519.074463] LNetError: 311775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8519.083324] LNet: Removed LNI 192.168.201.106@tcp [ 8519.405096] Key type .llcrypt unregistered [ 8519.406376] Key type ._llcrypt unregistered [ 8529.516903] Key type ._llcrypt registered [ 8529.518539] Key type .llcrypt registered [ 8529.558310] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 8535.788866] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 8536.142490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8536.196913] alg: No test for adler32 (adler32-zlib) [ 8537.059521] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 8537.142892] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 8538.719163] Key type lgssc registered [ 8539.075749] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8550.693803] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8550.700061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8551.781539] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8551.790659] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8551.825288] Lustre: lustre-MDT0000: new disk, initializing [ 8551.845633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8551.850895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8553.230152] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8557.307536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8557.350638] Lustre: 315602:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8557.373235] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8557.374765] Lustre: Skipped 1 previous similar message [ 8557.411713] Lustre: lustre-MDT0001: new disk, initializing [ 8557.443710] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8557.456330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8557.459676] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8558.791426] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8562.456343] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8564.327982] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8566.486720] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8566.583810] Lustre: lustre-OST0000: new disk, initializing [ 8566.585367] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8566.587567] Lustre: 317571:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8566.618175] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8568.190035] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8568.194159] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8568.227527] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8568.513488] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8572.099991] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8629.328249] Lustre: Mounted lustre-client [ 8629.653078] Lustre: Unmounted lustre-client [ 8634.339663] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8634.344283] Lustre: lustre-OST0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 8634.346357] Lustre: Skipped 1 previous similar message [ 8638.669046] Lustre: server umount lustre-OST0000 complete [ 8644.577553] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.201.106@tcp failed: rc = -107 [ 8644.577606] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8644.578370] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 8644.586813] Lustre: Skipped 2 previous similar messages [ 8649.696472] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 8649.698634] Lustre: Skipped 2 previous similar messages [ 8653.792793] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 8653.794959] Lustre: Skipped 1 previous similar message [ 8654.303109] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8654.362849] Lustre: server umount lustre-MDT0000 complete [ 8655.562146] LustreError: 317570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784946491 with bad export cookie 17659299601365015192 [ 8655.565630] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8655.565847] LustreError: 317570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8655.673538] Lustre: server umount lustre-MDT0001 complete [ 8659.200856] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 8660.166936] Key type lgssc unregistered [ 8660.287268] LNet: 319741:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8660.289768] LNetError: 319741:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8705.381275] LNet: Removed LNI 192.168.201.106@tcp [ 8705.667166] Key type .llcrypt unregistered [ 8705.668112] Key type ._llcrypt unregistered [ 8711.435394] Key type ._llcrypt registered [ 8711.436342] Key type .llcrypt registered [ 8711.475767] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 8717.797839] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 8718.136705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8718.193556] alg: No test for adler32 (adler32-zlib) [ 8719.045555] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 8719.130664] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 8720.711134] Key type lgssc registered [ 8721.083428] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8730.960377] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 22:29:26 (1784946566) [ 8733.100413] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8733.107383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8734.187737] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8734.196760] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8734.230433] Lustre: lustre-MDT0000: new disk, initializing [ 8734.252682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8734.257910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8735.570829] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8739.688384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8739.738974] Lustre: 322936:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8739.762106] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8739.764792] Lustre: Skipped 1 previous similar message [ 8739.798256] Lustre: lustre-MDT0001: new disk, initializing [ 8739.828524] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8739.840204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8739.844574] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8741.165691] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8744.670048] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8746.401716] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8748.662187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8748.756161] Lustre: lustre-OST0000: new disk, initializing [ 8748.757562] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8748.759788] Lustre: 324903:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8748.788033] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8750.405383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8750.409339] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8750.456481] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8750.780316] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8754.454800] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8760.803822] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8760.804750] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.201.106@tcp failed: rc = -107 [ 8760.805745] Lustre: lustre-OST0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 8760.808265] Lustre: Skipped 1 previous similar message [ 8762.063538] Lustre: server umount lustre-OST0000 complete [ 8765.408320] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.201.106@tcp failed: rc = -107 [ 8765.412341] LustreError: Skipped 1 previous similar message [ 8765.414269] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8765.425957] Lustre: server umount lustre-MDT0000 complete [ 8766.625792] LustreError: 322929:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784946602 with bad export cookie 995894036801799087 [ 8766.630702] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8766.755391] Lustre: server umount lustre-MDT0001 complete [ 8770.419238] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 8771.452786] Key type lgssc unregistered [ 8771.585387] LNet: 326807:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8771.587500] LNetError: 326807:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8771.593373] LNet: Removed LNI 192.168.201.106@tcp [ 8771.933374] Key type .llcrypt unregistered [ 8771.934981] Key type ._llcrypt unregistered [ 8779.726845] Key type ._llcrypt registered [ 8779.727714] Key type .llcrypt registered [ 8779.762479] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 8780.072293] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8780.079069] alg: No test for adler32 (adler32-zlib) [ 8780.932531] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 8781.014864] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 8782.591112] Key type lgssc registered [ 8782.936595] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8785.373050] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8785.378591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8786.542797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8787.862800] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8790.845488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8790.978780] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8792.328079] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8794.727044] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8796.565837] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8798.782056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8798.910121] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8800.778478] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8803.216150] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8817.235501] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8817.770461] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8821.730846] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.201.106@tcp failed: rc = -107 [ 8821.731457] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8821.732200] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 8821.741731] Lustre: Skipped 2 previous similar messages [ 8824.162350] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.6@tcp (stopping) [ 8824.164350] Lustre: Skipped 2 previous similar messages [ 8825.366191] Lustre: server umount lustre-MDT0000 complete [ 8826.625928] LustreError: 328039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784946662 with bad export cookie 17201015894574968261 [ 8826.629310] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8826.630539] LustreError: 328039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8826.848345] LustreError: 330554:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8826.848855] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8826.852829] LustreError: 330554:0:(ldlm_lib.c:1215:target_handle_connect()) Skipped 2 previous similar messages [ 8826.867201] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.106@tcp (stopping) [ 8829.275793] LustreError: 328082:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8829.280744] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.6@tcp (stopping) [ 8831.968568] LustreError: 328054:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8832.804529] Lustre: server umount lustre-MDT0001 complete [ 8834.401361] Lustre: lustre-OST0000: Not available for connect from 192.168.201.6@tcp (stopping) [ 8834.404204] Lustre: Skipped 1 previous similar message [ 8844.635526] Lustre: lustre-OST0000: Not available for connect from 192.168.201.6@tcp (stopping) [ 8844.637769] Lustre: Skipped 1 previous similar message [ 8848.351108] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8848.410822] Lustre: server umount lustre-OST0000 complete [ 8851.328242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8852.883167] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8853.924791] Lustre: 332255:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.106@tcp [ 8854.652570] Lustre: server umount MGS complete [ 8857.516375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8857.679071] LustreError: 332826:0:(ldlm_lib.c:1215:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.106@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8857.714790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8859.439152] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8862.570245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8862.739834] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8864.381748] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8867.213260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8869.216321] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8871.578475] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8875.147042] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 8876.938276] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8880.523860] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 8883.682952] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.201.106@tcp failed: rc = -107 [ 8883.683571] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8883.684293] Lustre: lustre-OST0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 8883.684299] Lustre: Skipped 1 previous similar message [ 8883.692071] Lustre: Skipped 1 previous similar message [ 8887.690056] Lustre: server umount lustre-OST0000 complete [ 8893.408082] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.201.106@tcp failed: rc = -107 [ 8893.410444] LustreError: Skipped 1 previous similar message [ 8893.411901] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8895.060575] Lustre: server umount lustre-MDT0000 complete [ 8896.310166] LustreError: 332852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784946732 with bad export cookie 17201015894574969087 [ 8896.312915] LustreError: 332852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8896.313885] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8896.428184] Lustre: server umount lustre-MDT0001 complete [ 8899.888551] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 8900.922773] Key type lgssc unregistered [ 8901.051562] LNet: 336261:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8901.055315] LNetError: 336261:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8901.065497] LNet: Removed LNI 192.168.201.106@tcp [ 8901.415088] Key type .llcrypt unregistered [ 8901.415991] Key type ._llcrypt unregistered [ 8907.448707] Key type ._llcrypt registered [ 8907.450533] Key type .llcrypt registered [ 8907.493441] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 8914.389439] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 8914.808462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8914.842283] alg: No test for adler32 (adler32-zlib) [ 8915.705330] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 8915.791604] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 8917.375178] Key type lgssc registered [ 8917.773230] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8930.159125] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8930.165828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8931.256465] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8931.266771] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8931.309093] Lustre: lustre-MDT0000: new disk, initializing [ 8931.332110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8931.338515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8932.740034] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8937.014900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8937.053179] Lustre: 339410:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8937.076505] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8937.078500] Lustre: Skipped 1 previous similar message [ 8937.116389] Lustre: lustre-MDT0001: new disk, initializing [ 8937.150132] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8937.163901] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8937.168301] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8938.444223] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8942.014372] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8943.860745] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8946.024844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8946.125472] Lustre: lustre-OST0000: new disk, initializing [ 8946.127329] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8946.130205] Lustre: 341377:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8946.160273] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8947.990229] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8949.947253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8949.951761] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8949.985099] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8951.888230] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8953.292919] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8953.367565] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8954.670779] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8954.738885] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8955.367347] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 192.168.201.106@tcp failed: rc = -107 [ 8955.367411] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8955.374445] Lustre: lustre-OST0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 8960.483158] Lustre: lustre-OST0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 8960.484668] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 192.168.201.106@tcp failed: rc = -107 [ 8960.486345] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8961.502983] Lustre: server umount lustre-OST0000 complete [ 8963.040460] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.201.106@tcp failed: rc = -107 [ 8963.043930] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8963.051208] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 8963.055756] Lustre: Skipped 1 previous similar message [ 8965.600982] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 8965.601627] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 8970.721399] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 8970.725251] Lustre: Skipped 2 previous similar messages [ 8977.375123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8977.437414] Lustre: server umount lustre-MDT0000 complete [ 8978.811338] LustreError: 339401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784946815 with bad export cookie 1681823024404867020 [ 8978.814904] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8978.816187] LustreError: 339401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8978.933947] Lustre: server umount lustre-MDT0001 complete [ 8981.891032] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 22:33:37 (1784946817) [ 8988.248484] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 8991.530642] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 9002.920495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9003.005953] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9003.018211] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9003.054144] Lustre: lustre-MDT0000: new disk, initializing [ 9003.075324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9003.082844] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9004.527535] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9008.607544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9008.648604] Lustre: 346128:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9008.668727] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9008.670410] Lustre: Skipped 1 previous similar message [ 9008.704985] Lustre: lustre-MDT0001: new disk, initializing [ 9008.737671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9008.750939] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9008.754064] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9010.054494] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9013.790458] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9015.800541] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9018.331595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9018.454574] Lustre: lustre-OST0000: new disk, initializing [ 9018.456066] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9018.458568] Lustre: 348098:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9018.493426] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9019.274080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9019.278490] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9019.325234] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9020.767407] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9024.605580] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9037.205712] Lustre: server umount lustre-OST0000 complete [ 9039.328352] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.201.106@tcp failed: rc = -107 [ 9039.330825] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 9039.335226] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 9039.337159] Lustre: Skipped 2 previous similar messages [ 9048.073551] Lustre: server umount lustre-MDT0000 complete [ 9049.349219] LustreError: 346923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784946885 with bad export cookie 1681823024404867909 [ 9049.352933] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9049.355120] LustreError: 346923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9058.829104] Lustre: server umount lustre-MDT0001 complete [ 9062.858891] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 9064.192466] Key type lgssc unregistered [ 9064.356840] LNet: 350247:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9064.362249] LNetError: 350247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9064.371637] LNet: Removed LNI 192.168.201.106@tcp [ 9064.845073] Key type .llcrypt unregistered [ 9064.847334] Key type ._llcrypt unregistered [ 9101.557073] Key type ._llcrypt registered [ 9101.558045] Key type .llcrypt registered [ 9101.593710] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 9101.691294] Key type .llcrypt unregistered [ 9101.692497] Key type ._llcrypt unregistered [ 9131.272309] Key type ._llcrypt registered [ 9131.273132] Key type .llcrypt registered [ 9131.314717] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 9131.463368] Key type .llcrypt unregistered [ 9131.465535] Key type ._llcrypt unregistered [ 9142.858387] Key type ._llcrypt registered [ 9142.859443] Key type .llcrypt registered [ 9142.904486] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 9143.034274] Key type .llcrypt unregistered [ 9143.035558] Key type ._llcrypt unregistered [ 9154.562299] Key type ._llcrypt registered [ 9154.563246] Key type .llcrypt registered [ 9154.599155] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 9154.718377] Key type .llcrypt unregistered [ 9154.719666] Key type ._llcrypt unregistered [ 9183.063252] Key type ._llcrypt registered [ 9183.065787] Key type .llcrypt registered [ 9183.106497] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 9183.211477] Key type .llcrypt unregistered [ 9183.212713] Key type ._llcrypt unregistered [ 9195.502623] Key type ._llcrypt registered [ 9195.503760] Key type .llcrypt registered [ 9195.547782] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 9195.653214] Key type .llcrypt unregistered [ 9195.654471] Key type ._llcrypt unregistered [ 9206.780790] Key type ._llcrypt registered [ 9206.782039] Key type .llcrypt registered [ 9206.826202] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 9206.948412] Key type .llcrypt unregistered [ 9206.950039] Key type ._llcrypt unregistered [ 9218.110055] Key type ._llcrypt registered [ 9218.111576] Key type .llcrypt registered [ 9218.152186] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 9218.267565] Key type .llcrypt unregistered [ 9218.269194] Key type ._llcrypt unregistered [ 9229.292405] Key type ._llcrypt registered [ 9229.293457] Key type .llcrypt registered [ 9229.331715] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 9229.443379] Key type .llcrypt unregistered [ 9229.444617] Key type ._llcrypt unregistered [ 9240.854956] Key type ._llcrypt registered [ 9240.856009] Key type .llcrypt registered [ 9240.893362] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 9241.017392] Key type .llcrypt unregistered [ 9241.018522] Key type ._llcrypt unregistered [ 9264.642317] Key type ._llcrypt registered [ 9264.643444] Key type .llcrypt registered [ 9264.682723] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 9264.775352] Key type .llcrypt unregistered [ 9264.776719] Key type ._llcrypt unregistered [ 9277.312777] Key type ._llcrypt registered [ 9277.313864] Key type .llcrypt registered [ 9277.356799] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 9277.465182] Key type .llcrypt unregistered [ 9277.466199] Key type ._llcrypt unregistered [ 9289.619859] Key type ._llcrypt registered [ 9289.621080] Key type .llcrypt registered [ 9289.658089] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 9289.752476] Key type .llcrypt unregistered [ 9289.754024] Key type ._llcrypt unregistered [ 9310.689177] Key type ._llcrypt registered [ 9310.690129] Key type .llcrypt registered [ 9310.724135] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 9317.990406] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 9318.399120] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9318.427756] alg: No test for adler32 (adler32-zlib) [ 9319.285228] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 9319.376556] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 9320.959178] Key type lgssc registered [ 9321.318865] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9336.812137] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9336.820406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9337.931653] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 9337.947089] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 9337.995755] Lustre: tst250fs-MDT0000: new disk, initializing [ 9338.020451] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9338.028553] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9339.544091] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9343.819629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9343.869569] Lustre: 370149:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9343.892530] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 9343.894472] Lustre: Skipped 1 previous similar message [ 9343.932546] Lustre: tst250fs-MDT0001: new disk, initializing [ 9343.965038] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9343.978187] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9343.981701] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9345.574147] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9348.206320] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9350.483234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9350.584918] Lustre: tst250fs-OST0000: new disk, initializing [ 9350.586538] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 9350.589217] Lustre: 371781:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9350.622078] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9352.616096] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9353.340943] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9353.344560] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9353.384849] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9356.700916] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9356.762869] Lustre: tst250fs-OST0001: new disk, initializing [ 9356.765634] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 9356.768773] Lustre: 372653:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9356.804727] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9358.797293] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9360.451800] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9360.455139] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9360.488847] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9361.405041] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 9374.689309] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.201.106@tcp failed: rc = -107 [ 9374.693619] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 9374.700933] Lustre: tst250fs-MDT0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 9375.715500] Lustre: tst250fs-MDT0000: Not available for connect from 192.168.201.106@tcp (stopping) [ 9375.716857] Lustre: tst250fs-MDT0000-lwp-OST0001: Connection to tst250fs-MDT0000 (at 192.168.201.106@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 9375.717768] Lustre: Skipped 1 previous similar message [ 9375.721697] Lustre: Skipped 1 previous similar message [ 9377.500912] Lustre: server umount tst250fs-MDT0000 complete [ 9378.703549] LustreError: 373266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1784947215 with bad export cookie 8089673361333476358 [ 9378.709367] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9378.837374] Lustre: server umount tst250fs-MDT0001 complete [ 9390.020568] Lustre: server umount tst250fs-OST0000 complete [ 9401.715182] Lustre: server umount tst250fs-OST0001 complete [ 9408.446063] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 9411.621713] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 9422.443807] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 22:40:57 (1784947257) [ 9422.948295] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 9428.360031] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 9431.838570] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 9442.037300] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 9141 sec ========== 22:41:17 (1784947277) [ 9442.575280] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 22:41:18 (1784947278) === [ 9443.781431] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 22:41:19 (1784947279) === [ 9460.236470] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 9461.586197] Key type lgssc unregistered [ 9461.765616] LNet: 381762:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9461.769477] LNetError: 381762:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9461.777497] LNet: Removed LNI 192.168.201.106@tcp [ 9462.260133] Key type .llcrypt unregistered [ 9462.262478] Key type ._llcrypt unregistered