[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 1609612862 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002000] x2apic enabled [ 0.002000] Switched APIC routing to physical x2apic. [ 0.002000] kvm-guest: setup PV IPIs [ 0.002000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.002000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.002000] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.002000] pid_max: default: 32768 minimum: 301 [ 0.002000] LSM: Security Framework initializing [ 0.002000] Yama: becoming mindful. [ 0.002000] SELinux: Initializing. [ 0.002000] *** VALIDATE selinux *** [ 0.002475] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.003000] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.003000] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.003000] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.003000] *** VALIDATE tmpfs *** [ 0.003000] *** VALIDATE proc *** [ 0.003000] *** VALIDATE cgroup *** [ 0.003000] *** VALIDATE cgroup2 *** [ 0.003000] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.003000] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.003000] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.003000] Spectre V2 : User space: Vulnerable [ 0.003000] Speculative Store Bypass: Vulnerable [ 0.003000] debug: unmapping init [mem 0xffffffff98859000-0xffffffff98860fff] [ 0.003000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.003695] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.004026] ... version: 2 [ 0.005015] ... bit width: 48 [ 0.006013] ... generic registers: 4 [ 0.007011] ... value mask: 0000ffffffffffff [ 0.008013] ... max period: 00007fffffffffff [ 0.009015] ... fixed-purpose events: 3 [ 0.010012] ... event mask: 000000070000000f [ 0.011281] rcu: Hierarchical SRCU implementation. [ 0.013496] smp: Bringing up secondary CPUs ... [ 0.014589] x86: Booting SMP configuration: [ 0.015042] .... node #0, CPUs: #1 #2 #3 [ 0.024190] smp: Brought up 1 node, 4 CPUs [ 0.026014] smpboot: Max logical packages: 1 [ 0.027018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.076446] node 0 deferred pages initialised in 47ms [ 0.082402] devtmpfs: initialized [ 0.083945] x86/mm: Memory block size: 128MB [ 0.086481] gcov: version magic: 0x41383552 [ 0.088181] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.089095] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.090230] pinctrl core: initialized pinctrl subsystem [ 0.091203] [ 0.091925] ************************************************************* [ 0.092015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.093014] ** ** [ 0.094012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.095014] ** ** [ 0.096012] ** This means that this kernel is built to expose internal ** [ 0.097015] ** IOMMU data structures, which may compromise security on ** [ 0.098013] ** your system. ** [ 0.099014] ** ** [ 0.100014] ** If you see this message and you are not debugging the ** [ 0.101020] ** kernel, report this immediately to your vendor! ** [ 0.102015] ** ** [ 0.103017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.104016] ************************************************************* [ 0.105727] NET: Registered protocol family 16 [ 0.106672] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.107077] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.108088] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.109677] cpuidle: using governor menu [ 0.112265] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.113753] PCI: Using configuration type 1 for base access [ 0.114183] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.121172] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.123017] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.127054] cryptd: max_cpu_qlen set to 1000 [ 0.130316] ACPI: Added _OSI(Module Device) [ 0.131013] ACPI: Added _OSI(Processor Device) [ 0.133013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.135012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.140336] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.147866] ACPI: Interpreter enabled [ 0.148056] ACPI: PM: (supports S0 S3 S4 S5) [ 0.149012] ACPI: Using IOAPIC for interrupt routing [ 0.150102] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.151386] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.161679] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.162044] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.163018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.164077] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.166579] acpiphp: Slot [2] registered [ 0.167216] acpiphp: Slot [5] registered [ 0.168201] acpiphp: Slot [6] registered [ 0.169121] acpiphp: Slot [7] registered [ 0.170147] acpiphp: Slot [8] registered [ 0.171121] acpiphp: Slot [9] registered [ 0.172130] acpiphp: Slot [10] registered [ 0.173131] acpiphp: Slot [3] registered [ 0.174126] acpiphp: Slot [4] registered [ 0.175101] acpiphp: Slot [11] registered [ 0.176195] acpiphp: Slot [12] registered [ 0.177114] acpiphp: Slot [13] registered [ 0.178140] acpiphp: Slot [14] registered [ 0.179167] acpiphp: Slot [15] registered [ 0.180134] acpiphp: Slot [16] registered [ 0.181198] acpiphp: Slot [17] registered [ 0.182278] acpiphp: Slot [18] registered [ 0.183614] acpiphp: Slot [19] registered [ 0.184181] acpiphp: Slot [20] registered [ 0.185422] acpiphp: Slot [21] registered [ 0.186318] acpiphp: Slot [22] registered [ 0.187137] acpiphp: Slot [23] registered [ 0.188172] acpiphp: Slot [24] registered [ 0.189147] acpiphp: Slot [25] registered [ 0.190081] acpiphp: Slot [26] registered [ 0.191162] acpiphp: Slot [27] registered [ 0.192117] acpiphp: Slot [28] registered [ 0.193113] acpiphp: Slot [29] registered [ 0.194124] acpiphp: Slot [30] registered [ 0.195104] acpiphp: Slot [31] registered [ 0.196061] PCI host bridge to bus 0000:00 [ 0.197023] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.198026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.199031] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.200033] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.201019] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.202024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.203182] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.205110] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.207357] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.213782] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.218061] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.219024] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.220025] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.221032] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.222780] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.224019] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.225044] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.226992] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.229018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.236018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.238976] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.242459] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.245019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.248017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.255014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.264359] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.268013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.271018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.278022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.286549] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.290015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.293013] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.300016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.308274] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.311016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.314022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.322018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.329549] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.332016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.335014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.342016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.349514] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.353014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.356018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.363015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.371384] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.373388] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.374397] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.375404] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.376306] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.380248] iommu: Default domain type: Passthrough [ 0.382477] SCSI subsystem initialized [ 0.384158] ACPI: bus type USB registered [ 0.385152] usbcore: registered new interface driver usbfs [ 0.386130] usbcore: registered new interface driver hub [ 0.387265] usbcore: registered new device driver usb [ 0.388220] pps_core: LinuxPPS API ver. 1 registered [ 0.389015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.390063] PTP clock support registered [ 0.393106] EDAC MC: Ver: 3.0.0 [ 0.395140] PCI: Using ACPI for IRQ routing [ 0.399011] NetLabel: Initializing [ 0.400010] NetLabel: domain hash size = 128 [ 0.401015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.402110] NetLabel: unlabeled traffic allowed by default [ 0.403849] vgaarb: loaded [ 0.406306] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.407025] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.422189] clocksource: Switched to clocksource kvm-clock [ 1.233934] VFS: Disk quotas dquot_6.6.0 [ 1.236164] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.251883] *** VALIDATE ramfs *** [ 1.255869] *** VALIDATE hugetlbfs *** [ 1.264912] pnp: PnP ACPI init [ 1.285850] pnp: PnP ACPI: found 6 devices [ 1.378569] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.395333] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.409204] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.427201] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.443766] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.451165] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.486929] NET: Registered protocol family 2 [ 1.489473] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.528534] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.548909] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.576038] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.593219] TCP: Hash tables configured (established 65536 bind 65536) [ 1.613754] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.639887] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.670111] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.688239] NET: Registered protocol family 1 [ 1.706561] RPC: Registered named UNIX socket transport module. [ 1.713558] RPC: Registered udp transport module. [ 1.724025] RPC: Registered tcp transport module. [ 1.735549] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.738293] NET: Registered protocol family 44 [ 1.739799] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.749627] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.751514] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.761044] PCI: CLS 0 bytes, default 64 [ 1.768432] Unpacking initramfs... [ 7.592124] debug: unmapping init [mem 0xffff9957fcc54000-0xffff9957fffbffff] [ 7.651450] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 7.653778] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 7.674239] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 8.903017] hrtimer: interrupt took 3252102 ns [ 10.654769] Initialise system trusted keyrings [ 10.657368] Key type blacklist registered [ 10.659765] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 10.680467] zbud: loaded [ 10.689557] *** VALIDATE nfs *** [ 10.690755] *** VALIDATE nfs4 *** [ 10.696892] pstore: using deflate compression [ 10.706601] Platform Keyring initialized [ 10.982057] NET: Registered protocol family 38 [ 10.985501] Key type asymmetric registered [ 10.987018] Asymmetric key parser 'x509' registered [ 11.009040] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 11.031342] io scheduler mq-deadline registered [ 11.033086] io scheduler kyber registered [ 11.045358] io scheduler bfq registered [ 11.054697] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 11.075959] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 11.091869] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 11.105265] ACPI: Power Button [PWRF] [ 11.153626] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 11.209622] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 11.255318] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 11.287705] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 11.355042] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 11.419744] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 11.496269] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 11.525367] Non-volatile memory driver v1.3 [ 11.539949] Linux agpgart interface v0.103 [ 11.752142] virtio_blk virtio1: [vda] 133944 512-byte logical blocks (68.6 MB/65.4 MiB) [ 11.771331] vda: detected capacity change from 0 to 68579328 [ 11.808318] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 11.811629] vdb: detected capacity change from 0 to 1073741824 [ 11.830419] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 11.834147] vdc: detected capacity change from 0 to 2621440000 [ 11.849620] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 11.853216] vdd: detected capacity change from 0 to 2621440000 [ 11.867729] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 11.870872] vde: detected capacity change from 0 to 4294967296 [ 11.889346] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 11.892629] vdf: detected capacity change from 0 to 4294967296 [ 11.898799] libphy: Fixed MDIO Bus: probed [ 11.952728] usbcore: registered new interface driver usbserial_generic [ 11.973616] usbserial: USB Serial support registered for generic [ 11.987126] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 11.999829] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 12.013800] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 12.019733] mousedev: PS/2 mouse device common for all mice [ 12.029454] rtc_cmos 00:05: RTC can wake from S4 [ 12.045467] rtc_cmos 00:05: registered as rtc0 [ 12.051781] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 12.052192] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 12.052299] intel_pstate: CPU model not supported [ 12.087106] hid: raw HID events driver (C) Jiri Kosina [ 12.117078] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 12.128759] usbcore: registered new interface driver usbhid [ 12.130963] usbhid: USB HID core driver [ 12.140153] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 12.150059] drop_monitor: Initializing network drop monitor service [ 12.150786] Initializing XFRM netlink socket [ 12.157025] NET: Registered protocol family 10 [ 12.161648] Segment Routing with IPv6 [ 12.193513] NET: Registered protocol family 17 [ 12.215686] mpls_gso: MPLS GSO support [ 12.231790] RAS: Correctable Errors collector initialized. [ 12.239456] AVX version of gcm_enc/dec engaged. [ 12.242700] AES CTR mode by8 optimization enabled [ 12.610816] sched_clock: Marking stable (12610757681, 0)->(15033599314, -2422841633) [ 12.640379] registered taskstats version 1 [ 12.656179] Loading compiled-in X.509 certificates [ 12.664956] zswap: loaded using pool lzo/zbud [ 12.812919] Key type big_key registered [ 12.887587] Key type encrypted registered [ 12.899105] ima: No TPM chip found, activating TPM-bypass! [ 12.901685] ima: Allocated hash algorithm: sha1 [ 12.921910] ima: No architecture policies found [ 12.926527] evm: Initialising EVM extended attributes: [ 12.927897] evm: security.selinux [ 12.930532] evm: security.ima [ 12.933712] evm: security.capability [ 12.938161] evm: HMAC attrs: 0x1 [ 12.944320] rtc_cmos 00:05: setting system clock to 2026-04-10 14:38:59 UTC (1775831939) [ 12.959466] debug: unmapping init [mem 0xffffffff99803000-0xffffffff999fffff] [ 12.983227] debug: unmapping init [mem 0xffffffff98582000-0xffffffff98858fff] [ 13.002113] Write protecting the kernel read-only data: 28672k [ 13.013555] debug: unmapping init [mem 0xffffffff96c03000-0xffffffff96dfffff] [ 13.037702] debug: unmapping init [mem 0xffffffff97514000-0xffffffff975fffff] [ 13.220978] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 13.241931] systemd[1]: Detected virtualization kvm. [ 13.253828] systemd[1]: Detected architecture x86-64. [ 13.262708] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 13.437245] systemd[1]: No hostname configured. [ 13.441705] systemd[1]: Set hostname to . [ 13.463400] random: systemd: uninitialized urandom read (16 bytes read) [ 13.474087] systemd[1]: Initializing machine ID from random generator. [ 13.690662] random: ln: uninitialized urandom read (6 bytes read) [ 14.177288] random: systemd: uninitialized urandom read (16 bytes read) [ 14.197239] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 14.243229] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 14.271536] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Sockets. [ OK ] Reached target Timers. Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 18.274700] device-mapper: uevent: version 1.0.3 [ 18.303639] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 22.916649] virtio_net virtio0 ens2: renamed from eth0 [ 23.076908] random: fast init done [ 23.555290] scsi host0: ata_piix [ 23.637291] scsi host1: ata_piix [ 23.640864] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 23.646044] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 29.538060] random: crng init done [ 29.556370] random: 7 urandom warning(s) missed due to ratelimiting [ 33.314782] dracut-initqueue[593]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 35.816213] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 40.658147] printk: systemd: 26 output lines suppressed due to ratelimiting [ 41.590224] SELinux: Disabled at runtime. [ 41.679653] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 41.691498] systemd[1]: Detected virtualization kvm. [ 41.693390] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 44.543182] systemd[1]: initrd-switch-root.service: Succeeded. [ 44.557703] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 44.595726] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 44.611468] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 44.629403] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 44.684865] systemd[1]: Starting Journal Service... Starting Journal Service... [ 44.707975] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on Process Core Dump Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ 45.032415] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Stopped target Initrd Root File System. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Remount Root and Kernel File Systems... [ OK ] Listening on RPCbind Server Activation Socket. Mounting Kernel Debug File System... Mounting Huge Pages File System... Starting udev Coldplug all Devices... [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Paths. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 47.348738] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 48.655065] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 48.690909] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 49.537302] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 49.649562] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (9s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit) [ ***] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (12s / no limit) [ *] A start job is running for Configur…only root support (12s / no limit) [ **] A start job is running for Configur…only root support (13s / no limit) [ ***] A start job is running for Configur…only root support (13s / no limit) [ *** ] A start job is running for Configur…only root support (13s / no limit)[ 58.201447] Key type dns_resolver registered [ *** ] A start job is running for Configur…only root support (14s / no limit) [*** ] A start job is running for Configur…only root support (14s / no limit)[ 59.364331] NFS: Registering the id_resolver key type [ 59.367632] Key type id_resolver registered [ 59.370675] Key type id_legacy registered [** ] A start job is running for Configur…only root support (15s / no limit) [* ] A start job is running for Configur…only root support (15s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started dnf makecache --timer. Starting Login Service... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started D-Bus System Message Bus. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg320-server login: [ 118.065290] libcfs: loading out-of-tree module taints kernel. [ 118.094240] Key type ._llcrypt registered [ 118.097624] Key type .llcrypt registered [ 118.174829] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 136.865461] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 138.528604] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 138.557756] alg: No test for adler32 (adler32-zlib) [ 140.046759] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 140.734360] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 142.511263] Key type lgssc registered [ 144.385694] Lustre: Echo OBD driver; http://www.lustre.org/ [ 158.999722] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 197.298503] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 208.522546] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 208.562941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 209.851199] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 209.883991] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 209.958763] Lustre: lustre-MDT0000: new disk, initializing [ 210.028775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 210.041460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 213.510808] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 226.632308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 226.716970] Lustre: 6508:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 226.746902] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 226.750681] Lustre: Skipped 1 previous similar message [ 226.848756] Lustre: lustre-MDT0001: new disk, initializing [ 226.929516] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 226.969838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 226.982076] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 230.893407] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 235.368737] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 244.711102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 244.954543] Lustre: lustre-OST0000: new disk, initializing [ 244.959535] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 245.012375] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 250.683282] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 252.987946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 252.999862] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 253.122292] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 265.627485] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 265.757702] Lustre: lustre-OST0001: new disk, initializing [ 265.771643] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 265.828091] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 271.527954] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 273.474447] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 273.494158] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 273.567970] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 283.382795] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 293.822710] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 304.994673] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing check_logdir /tmp/testlogs/ [ 309.454318] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing yml_node [ 314.079772] Lustre: DEBUG MARKER: Client: 2.17.51.75 [ 316.411724] Lustre: DEBUG MARKER: MDS: 2.17.51.75 [ 318.554189] Lustre: DEBUG MARKER: OSS: 2.17.51.75 [ 319.947472] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Apr 10 10:44:04 EDT 2026 [ 337.584669] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 339.203483] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 355.297620] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 355.308326] Lustre: Skipped 2 previous similar messages [ 355.311414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 360.423096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 360.432104] Lustre: Skipped 4 previous similar messages [ 361.611505] Lustre: server umount lustre-MDT0000 complete [ 364.973778] LustreError: 6502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775832291 with bad export cookie 145597970447736137 [ 364.978313] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 364.982151] LustreError: 6502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 365.536059] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 365.555636] Lustre: Skipped 1 previous similar message [ 365.560976] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 365.573466] Lustre: Skipped 1 previous similar message [ 369.571448] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 369.585495] Lustre: Skipped 1 previous similar message [ 374.689094] LustreError: 6515:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 374.692755] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 374.704458] LustreError: 6515:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 379.359368] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 379.763580] Lustre: server umount lustre-MDT0001 complete [ 383.619703] Lustre: server umount lustre-OST0000 complete [ 387.375508] Lustre: server umount lustre-OST0001 complete [ 393.509833] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 402.221315] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 432.047550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 432.304825] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 432.352392] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 432.427203] Lustre: lustre-MDT0000: new disk, initializing [ 432.511329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 432.525758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 435.945694] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 445.516777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 445.681226] Lustre: 14990:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 445.753798] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 445.757134] Lustre: Skipped 1 previous similar message [ 445.815372] Lustre: lustre-MDT0001: new disk, initializing [ 445.888498] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 445.914444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 445.933095] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 449.825612] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 456.142604] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 459.707849] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 466.083067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 466.342929] Lustre: lustre-OST0000: new disk, initializing [ 466.349454] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 466.444896] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 467.641896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 467.651916] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 467.769226] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 470.876814] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 477.237413] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 481.069976] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 481.235901] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 484.949640] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 485.108714] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 488.419682] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 488.420384] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 488.430391] LustreError: Skipped 1 previous similar message [ 488.439040] Lustre: Skipped 1 previous similar message [ 488.441598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 488.459980] Lustre: Skipped 2 previous similar messages [ 492.767811] Lustre: server umount lustre-OST0000 complete [ 497.121190] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 497.131598] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 497.145466] Lustre: Skipped 1 previous similar message [ 501.883633] Lustre: server umount lustre-MDT0000 complete [ 503.777594] LustreError: 14997:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 503.796036] LustreError: 14997:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 504.692117] LustreError: 15768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775832431 with bad export cookie 145597970447737670 [ 504.692910] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 504.707290] LustreError: 15768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 504.947980] Lustre: server umount lustre-MDT0001 complete [ 509.009766] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 10:47:13 (1775832433) [ 520.858430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 521.106890] LustreError: 19459:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 521.163288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 523.625821] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 526.310050] LustreError: 19460:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 528.590254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 531.352127] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 534.925949] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 537.163256] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 538.435963] Lustre: Failing over lustre-MDT0000 [ 538.563528] Lustre: server umount lustre-MDT0000 complete [ 539.104341] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 539.110205] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 539.118166] Lustre: Skipped 1 previous similar message [ 539.121665] LustreError: 20170:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 550.368306] LustreError: 20170:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 550.380400] LustreError: 20170:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 554.328964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 554.403502] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 555.617896] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 555.622372] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 555.637854] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 557.211892] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 562.622608] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 566.239990] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 566.246811] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 566.256257] Lustre: Skipped 1 previous similar message [ 566.261621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 566.267840] Lustre: Skipped 3 previous similar messages [ 571.892690] Lustre: server umount lustre-MDT0000 complete [ 574.277480] LustreError: 19439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775832500 with bad export cookie 145597970447739084 [ 574.281866] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 574.285269] LustreError: 19439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 574.483128] Lustre: server umount lustre-MDT0001 complete [ 580.346395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 580.640617] LustreError: 22454:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 583.069610] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 588.731857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 588.949803] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 588.959399] Lustre: Skipped 3 previous similar messages [ 591.643505] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 595.425915] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 597.952273] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 599.302504] Lustre: Failing over lustre-MDT0000 [ 599.518348] Lustre: server umount lustre-MDT0000 complete [ 599.521369] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 599.539679] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 599.550386] Lustre: Skipped 1 previous similar message [ 613.862734] LustreError: 22454:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 613.875056] LustreError: 22454:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 615.860072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 615.973106] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 616.165697] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 618.648133] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 621.537502] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 621.542175] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 621.549617] Lustre: Skipped 1 previous similar message [ 621.563210] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 623.628254] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 626.657151] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 626.673855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 626.680685] Lustre: Skipped 2 previous similar messages [ 632.519462] Lustre: server umount lustre-MDT0000 complete [ 634.969594] LustreError: 22436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775832561 with bad export cookie 145597970447740022 [ 634.974964] LustreError: 22436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 634.975168] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 635.159916] Lustre: server umount lustre-MDT0001 complete [ 640.583137] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 10:49:25 (1775832565) [ 650.321768] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 656.327307] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 680.178439] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 686.707369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 686.858065] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 686.880939] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 686.935838] Lustre: lustre-MDT0000: new disk, initializing [ 687.000587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 689.378675] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 696.143155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 696.218744] Lustre: 28429:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 696.397887] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 698.695262] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 703.517405] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 705.677656] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 709.918817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 711.743256] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 711.776703] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 711.785962] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 713.665470] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 718.513791] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 723.578255] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 736.373680] Lustre: server umount lustre-OST0000 complete [ 742.371545] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 742.373649] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 742.376930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 742.376937] Lustre: Skipped 2 previous similar messages [ 742.395343] Lustre: Skipped 4 previous similar messages [ 748.561443] Lustre: server umount lustre-MDT0000 complete [ 750.864550] LustreError: 28421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775832677 with bad export cookie 145597970447740428 [ 750.868830] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 750.869978] LustreError: 28421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 761.427493] Lustre: server umount lustre-MDT0001 complete [ 767.389800] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 768.894532] Key type lgssc unregistered [ 769.055781] LNet: 31999:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 769.060266] LNetError: 31999:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 769.072813] LNet: Removed LNI 192.168.203.120@tcp [ 769.512144] Key type .llcrypt unregistered [ 769.514527] Key type ._llcrypt unregistered [ 782.669216] Key type ._llcrypt registered [ 782.671052] Key type .llcrypt registered [ 782.729506] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 783.279091] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 783.292713] alg: No test for adler32 (adler32-zlib) [ 784.236715] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 784.350761] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 785.976755] Key type lgssc registered [ 786.556264] Lustre: Echo OBD driver; http://www.lustre.org/ [ 790.929978] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 790.941891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 792.198707] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 794.292978] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 798.533733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 798.711307] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 800.498217] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 803.190163] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 805.053101] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 808.811060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 808.939796] LustreError: 34965:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 808.952744] LustreError: 34965:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 809.037346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 811.892901] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 814.061077] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 814.534127] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 826.025945] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 10:52:30 (1775832750) [ 826.908838] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 827.877309] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 831.247725] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 10:52:36 (1775832756) [ 835.452325] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 10:52:40 (1775832760) [ 835.908331] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 839.813277] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 839.819466] Lustre: Skipped 2 previous similar messages [ 840.691924] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 10:52:45 (1775832765) [ 844.817338] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 844.820653] Lustre: Skipped 4 previous similar messages [ 852.375168] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 10:52:57 (1775832777) [ 853.812877] Lustre: *** cfs_fail_loc=131b, val=2*** [ 855.876378] Lustre: *** cfs_fail_loc=131b, val=2*** [ 860.256822] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 10:53:05 (1775832785) [ 861.244520] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 861.254473] Lustre: Skipped 9 previous similar messages [ 868.048708] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 10:53:12 (1775832792) [ 875.491080] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 875.496155] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 875.509575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 878.560621] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 878.564325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 878.576274] Lustre: Skipped 1 previous similar message [ 883.680276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 883.683850] Lustre: Skipped 1 previous similar message [ 888.804270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 888.809485] Lustre: Skipped 1 previous similar message [ 889.312284] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 889.395313] Lustre: server umount lustre-OST0000 complete [ 893.922068] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 893.930697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 893.936536] Lustre: Skipped 1 previous similar message [ 897.331311] Lustre: server umount lustre-MDT0000 complete [ 899.041556] LustreError: 33370:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 899.051508] LustreError: 33370:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 899.381158] LustreError: 33353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775832825 with bad export cookie 14784267999455379132 [ 899.384106] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 899.390857] LustreError: 33353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 899.564924] Lustre: server umount lustre-MDT0001 complete [ 905.016531] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 906.516026] Key type lgssc unregistered [ 906.699040] LNet: 40861:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 906.705139] LNetError: 40861:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 906.716642] LNet: Removed LNI 192.168.203.120@tcp [ 907.131573] Key type .llcrypt unregistered [ 907.133648] Key type ._llcrypt unregistered [ 916.512488] Key type ._llcrypt registered [ 916.514791] Key type .llcrypt registered [ 916.597740] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 925.260995] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 925.650657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 925.684091] alg: No test for adler32 (adler32-zlib) [ 926.584316] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 926.686558] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 928.279222] Key type lgssc registered [ 928.879738] Lustre: Echo OBD driver; http://www.lustre.org/ [ 945.405544] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 945.417912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 946.559406] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 946.585110] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 946.633585] Lustre: lustre-MDT0000: new disk, initializing [ 946.676589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 946.688605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 948.478457] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 953.901579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 953.969734] Lustre: 43955:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 954.014340] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 954.019934] Lustre: Skipped 1 previous similar message [ 954.066317] Lustre: lustre-MDT0001: new disk, initializing [ 954.128871] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 954.157631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 954.165038] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 956.046678] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 960.106214] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 962.086400] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 965.476738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 965.606493] Lustre: lustre-OST0000: new disk, initializing [ 965.610124] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 965.659864] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 968.539908] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 971.293837] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 971.299742] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 971.359205] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 972.626806] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 974.659664] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 974.762971] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 976.751232] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 976.855480] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 981.472311] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 981.476359] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 981.477741] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 981.484431] Lustre: Skipped 1 previous similar message [ 981.492627] Lustre: Skipped 1 previous similar message [ 986.595145] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 986.599991] Lustre: Skipped 1 previous similar message [ 991.712786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 991.717165] Lustre: Skipped 1 previous similar message [ 992.223129] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 992.291192] Lustre: server umount lustre-OST0000 complete [ 995.297363] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 995.302734] LustreError: Skipped 1 previous similar message [ 995.305577] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 995.317291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 996.839680] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1002.232847] Lustre: server umount lustre-MDT0000 complete [ 1003.902448] LustreError: 43946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775832930 with bad export cookie 10083839949950992842 [ 1003.905163] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1003.907738] LustreError: 43946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1004.045486] Lustre: server umount lustre-MDT0001 complete [ 1008.810609] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 10:55:33 (1775832933) [ 1012.071842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1012.349273] LustreError: 48003:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1012.362586] LustreError: 48003:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1012.387345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1014.370871] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1017.824517] LustreError: 48004:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1018.494768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1018.676491] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1020.459858] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1022.947518] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1024.556525] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1027.670499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1027.823420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1030.438183] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1032.999223] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1041.408486] LustreError: 50195:0:(llog.c:444:llog_init_handle()) llog type is not specified! [ 1041.825539] Lustre: Modifying parameter general.timeout=1 in log params [ 1043.138731] Lustre: Modifying parameter general.timeout=4 in log params [ 1043.142076] Lustre: Skipped 2 previous similar messages [ 1045.305679] Lustre: Modifying parameter general.timeout=9 in log params [ 1045.309046] Lustre: Skipped 4 previous similar messages [ 1049.454501] Lustre: Modifying parameter general.timeout=19 in log params [ 1049.460992] Lustre: Skipped 9 previous similar messages [ 1057.541901] Lustre: Modifying parameter general.timeout=37 in log params [ 1057.544660] Lustre: Skipped 17 previous similar messages [ 1073.835370] Lustre: Modifying parameter general.timeout=78 in log params [ 1073.839279] Lustre: Skipped 40 previous similar messages [ 1100.504867] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 10:57:05 (1775833025) [ 1101.885157] Lustre: 57103:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1103.647673] Lustre: 57316:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1103.656016] Lustre: 57316:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 1 previous similar message [ 1105.836931] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=5000 in log params [ 1105.840689] Lustre: Skipped 58 previous similar messages [ 1106.302790] Lustre: 57611:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1106.313293] Lustre: 57611:0:(mgs_llog.c:1348:mgs_modify_param()) Skipped 2 previous similar messages [ 1118.849487] Lustre: 58953:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1118.856777] LustreError: 58953:0:(mgs_handler.c:1164:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1120.760634] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 10:57:25 (1775833045) [ 1126.369966] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1126.373971] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1126.380904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1126.384256] Lustre: Skipped 2 previous similar messages [ 1128.370678] Lustre: server umount lustre-MDT0000 complete [ 1130.060668] LustreError: 54978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775833056 with bad export cookie 10083839949950993710 [ 1130.061052] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1130.068815] LustreError: 54978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1130.073466] LustreError: 47995:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 1132.245026] Lustre: server umount lustre-MDT0001 complete [ 1156.172701] Lustre: server umount lustre-OST0000 complete [ 1168.123955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1168.256628] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1168.375697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1170.033732] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1173.551436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1173.606770] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1173.620141] Lustre: 61479:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1175.478933] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1177.951582] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1179.611425] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1182.559985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1182.665815] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1182.718089] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 1182.721288] Lustre: Skipped 3 previous similar messages [ 1185.265880] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1187.553625] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1193.577387] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1193.585404] Lustre: Skipped 16 previous similar messages [ 1193.626191] Lustre: 63708:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1198.559995] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1198.563631] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1198.569664] Lustre: Skipped 2 previous similar messages [ 1199.075435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1199.079057] Lustre: Skipped 2 previous similar messages [ 1202.212027] Lustre: server umount lustre-OST0000 complete [ 1209.957908] Lustre: server umount lustre-MDT0000 complete [ 1211.463541] LustreError: 62205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775833138 with bad export cookie 10083839949950996174 [ 1211.467240] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1211.471409] LustreError: 62205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1211.608537] Lustre: server umount lustre-MDT0001 complete [ 1215.976340] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 1217.203666] Key type lgssc unregistered [ 1217.349558] LNet: 64898:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1217.353553] LNetError: 64898:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1217.361441] LNet: Removed LNI 192.168.203.120@tcp [ 1217.722310] Key type .llcrypt unregistered [ 1217.723690] Key type ._llcrypt unregistered [ 1228.486558] Key type ._llcrypt registered [ 1228.488071] Key type .llcrypt registered [ 1228.534050] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 1228.902701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1228.921503] alg: No test for adler32 (adler32-zlib) [ 1229.800428] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 1229.914775] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 1231.511236] Key type lgssc registered [ 1231.972241] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1235.187080] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1235.196292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1236.401102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1237.918946] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1241.380727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1241.641139] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1243.224346] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1245.339374] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1246.711183] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1249.299139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1249.507664] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1251.732500] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1253.726800] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1267.313047] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1271.266207] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1271.267504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1271.272946] Lustre: Skipped 2 previous similar messages [ 1271.277538] Lustre: Skipped 2 previous similar messages [ 1276.321072] Lustre: server umount lustre-MDT0000 complete [ 1276.385358] LustreError: 67457:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1276.394699] LustreError: 67457:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1277.845282] LustreError: 67910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775833204 with bad export cookie 395933305740230902 [ 1277.848128] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1277.850563] LustreError: 67910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1277.856334] LustreError: 66277:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 1277.983251] Lustre: server umount lustre-MDT0001 complete [ 1305.164378] Lustre: server umount lustre-OST0000 complete [ 1314.985709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1315.118991] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1315.138760] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1315.262121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1316.785386] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1319.970880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1320.017130] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1320.028724] Lustre: 70841:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1320.204889] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1321.664666] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1323.669570] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1324.979225] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1327.371141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1327.447221] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1329.663115] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1331.689163] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1332.253031] Lustre: 73076:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1332.922138] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1339.361127] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1339.365054] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1339.365835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1339.372109] Lustre: Skipped 1 previous similar message [ 1339.376988] Lustre: Skipped 1 previous similar message [ 1340.437391] Lustre: server umount lustre-OST0000 complete [ 1344.481144] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1344.482193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1344.486319] Lustre: Skipped 1 previous similar message [ 1344.490740] Lustre: Skipped 1 previous similar message [ 1349.599849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1349.602047] Lustre: Skipped 1 previous similar message [ 1354.719898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1354.722678] Lustre: Skipped 1 previous similar message [ 1356.255122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1356.317591] Lustre: server umount lustre-MDT0000 complete [ 1357.569996] LustreError: 70879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775833284 with bad export cookie 395933305740231700 [ 1357.572955] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1357.573261] LustreError: 70879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1357.684215] Lustre: server umount lustre-MDT0001 complete [ 1361.230951] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 1362.257522] Key type lgssc unregistered [ 1362.388418] LNet: 74277:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1362.392316] LNetError: 74277:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1362.406618] LNet: Removed LNI 192.168.203.120@tcp [ 1362.700729] Key type .llcrypt unregistered [ 1362.701986] Key type ._llcrypt unregistered [ 1371.665638] Key type ._llcrypt registered [ 1371.667124] Key type .llcrypt registered [ 1371.705645] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 1372.020225] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1372.051068] alg: No test for adler32 (adler32-zlib) [ 1372.899176] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 1372.978402] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 1374.559152] Key type lgssc registered [ 1374.890380] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1377.420772] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1377.428281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1378.599334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1379.926866] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1382.801303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1382.999029] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1384.273638] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1386.097125] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1387.368956] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1389.672274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1389.862638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1391.832295] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1393.660246] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1411.221504] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1411.817866] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1411.820112] Lustre: Skipped 1 previous similar message [ 1413.031129] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1413.034263] Lustre: Skipped 3 previous similar messages [ 1415.059299] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1415.061394] Lustre: Skipped 6 previous similar messages [ 1419.150543] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1419.153358] Lustre: Skipped 13 previous similar messages [ 1427.426462] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1427.428532] Lustre: Skipped 28 previous similar messages [ 1442.203535] Lustre: 83398:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1443.027441] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 11:02:48 (1775833368) [ 1443.494987] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1444.017817] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 11:02:49 (1775833369) [ 1445.346666] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1445.346757] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1445.351881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1445.355974] Lustre: Skipped 1 previous similar message [ 1450.464663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1450.467498] Lustre: Skipped 1 previous similar message [ 1455.585601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1459.167166] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1459.215314] Lustre: server umount lustre-OST0000 complete [ 1460.704377] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1460.705112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1460.708302] Lustre: Skipped 2 previous similar messages [ 1460.711359] Lustre: Skipped 2 previous similar messages [ 1465.824564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1465.826706] Lustre: Skipped 1 previous similar message [ 1466.519738] Lustre: server umount lustre-MDT0000 complete [ 1467.801084] LustreError: 77272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775833394 with bad export cookie 17523483552461961004 [ 1467.803401] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1467.805923] LustreError: 77272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1467.933560] Lustre: server umount lustre-MDT0001 complete [ 1471.613806] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 1472.649229] Key type lgssc unregistered [ 1472.772385] LNet: 84630:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1472.775366] LNetError: 84630:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1472.782292] LNet: Removed LNI 192.168.203.120@tcp [ 1473.066462] Key type .llcrypt unregistered [ 1473.068162] Key type ._llcrypt unregistered [ 1474.439890] Key type ._llcrypt registered [ 1474.441859] Key type .llcrypt registered [ 1474.482913] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1475.860894] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules [ 1476.274689] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1476.305171] alg: No test for adler32 (adler32-zlib) [ 1477.178063] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 1477.269324] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 1478.855149] Key type lgssc registered [ 1479.299890] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1479.666966] LustreError: 85749:0:(lu_object.c:1477:lu_context_key_register()) cfs_race id 60d sleeping [ 1486.770383] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 1489.794896] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1500.832728] LustreError: 85749:0:(lu_object.c:1477:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1500.842408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1502.016337] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1503.244336] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1505.899793] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 11:03:50 (1775833430) [ 1507.327241] LustreError: 86786:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1507.330991] LustreError: 86786:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1507.333748] LustreError: 86786:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 5, retries 0, failed: rc = -5 [ 1507.453846] Lustre: server umount lustre-MDT0000 complete [ 1511.413362] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 1512.425208] Key type lgssc unregistered [ 1512.562386] LNet: 88244:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1512.565521] LNetError: 88244:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1512.574333] LNet: Removed LNI 192.168.203.120@tcp [ 1512.899570] Key type .llcrypt unregistered [ 1512.900834] Key type ._llcrypt unregistered [ 1519.365718] Key type ._llcrypt registered [ 1519.366703] Key type .llcrypt registered [ 1519.408691] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 1519.676306] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1519.684280] alg: No test for adler32 (adler32-zlib) [ 1520.526966] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 1520.594748] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 1522.167131] Key type lgssc registered [ 1522.498529] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1524.850638] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1524.856394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1526.022711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1527.180357] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1529.786916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1529.971033] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1531.189927] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1532.808632] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1533.888285] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1535.941956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1536.110428] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1537.815794] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1539.413165] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1560.517379] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 11:04:45 (1775833485) [ 1561.046887] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1561.620297] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 11:04:46 (1775833486) [ 1565.663839] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1565.667183] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1565.672692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1566.176365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1566.178343] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1568.662380] Lustre: server umount lustre-MDT0000 complete [ 1569.849054] LustreError: 89402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775833496 with bad export cookie 16085488490349424 [ 1569.851616] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1569.852317] LustreError: 89402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1569.958488] Lustre: server umount lustre-MDT0001 complete [ 1581.440257] Lustre: server umount lustre-OST0000 complete [ 1585.238509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1585.388459] LustreError: 93354:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1585.394245] LustreError: 93354:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1585.439653] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1586.732573] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1589.523515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1589.764330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1591.114837] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1592.894167] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1594.014966] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1598.180059] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1598.256519] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1598.259421] LustreError: 93343:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1598.269229] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1598.272352] LustreError: 95210:0:(tgt_mount.c:2545:server_fill_super()) Unable to start targets: -98 [ 1598.275251] LustreError: 95210:0:(tgt_mount.c:2063:server_put_super()) no obd lustre-OST0000 [ 1598.278302] LustreError: 95210:0:(tgt_mount.c:114:server_deregister_mount()) lustre-OST0000 not registered [ 1598.305666] Lustre: server umount lustre-OST0000 complete [ 1598.307811] LustreError: 95210:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1601.025347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1601.097975] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1601.100963] LustreError: 93343:0:(mgs_handler.c:567:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1601.109943] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1601.112889] LustreError: 95675:0:(tgt_mount.c:2545:server_fill_super()) Unable to start targets: -98 [ 1601.115185] LustreError: 95675:0:(tgt_mount.c:2063:server_put_super()) no obd lustre-OST0000 [ 1601.117171] LustreError: 95675:0:(tgt_mount.c:114:server_deregister_mount()) lustre-OST0000 not registered [ 1601.137865] Lustre: server umount lustre-OST0000 complete [ 1601.139272] LustreError: 95675:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1604.280376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1604.346505] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1604.348581] Lustre: Found index 0 for lustre-OST0000, updating log [ 1604.350923] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1604.372270] Lustre: lustre-OST0000: new disk, initializing [ 1604.374353] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1604.460140] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1606.312849] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1607.512538] Lustre: Failing over lustre-OST0000 [ 1607.557512] Lustre: server umount lustre-OST0000 complete [ 1608.773427] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1608.774193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1608.777461] Lustre: Skipped 2 previous similar messages [ 1608.780576] Lustre: Skipped 3 previous similar messages [ 1613.792528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1613.794501] Lustre: Skipped 1 previous similar message [ 1614.940375] Lustre: server umount lustre-MDT0000 complete [ 1616.167160] LustreError: 95691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775833542 with bad export cookie 16085488490352154 [ 1616.170349] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1616.170808] LustreError: 95691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1616.289199] Lustre: server umount lustre-MDT0001 complete [ 1619.120257] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 11:05:44 (1775833544) [ 1624.439298] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 1628.087439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1628.220655] LustreError: 98509:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1628.247144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1629.544290] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1632.237855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1633.621988] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1634.521047] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 1636.533409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1638.443323] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1641.062940] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1641.093935] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 1641.130157] Lustre: lustre-OST0001: new disk, initializing [ 1641.131688] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1642.997753] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1644.220896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1644.224837] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1644.250679] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1646.013627] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1651.554488] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1659.022068] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.20@tcp (stopping) [ 1659.024879] Lustre: Skipped 2 previous similar messages [ 1659.874329] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1659.877664] Lustre: Skipped 1 previous similar message [ 1669.256330] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.20@tcp (stopping) [ 1669.260441] Lustre: Skipped 7 previous similar messages [ 1672.159166] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1672.227262] Lustre: server umount lustre-MDT0001 complete [ 1674.376071] LustreError: 99231:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1675.523190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1675.557408] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1675.560088] Lustre: Found index 1 for lustre-MDT0001, updating log [ 1675.577762] Lustre: 98498:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1675.848834] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1675.851020] Lustre: Skipped 3 previous similar messages [ 1677.157037] LustreError: 102470:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 1677.159612] LustreError: 102470:0:(tgt_mount.c:666:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 1677.164608] LustreError: 98501:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 1677.194273] LustreError: 102484:0:(obd_config.c:2101:class_config_llog_handler()) MGC192.168.203.120@tcp: cfg command failed: rc = -17 [ 1677.198701] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1677.198701] [ 1677.284935] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1680.610060] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 11:06:45 (1775833605) [ 1680.872607] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 1686.091828] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 1692.316201] Lustre: 104341:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1700.407440] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1701.665274] Lustre: 105472:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1752.543598] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1752.544547] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1752.545441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1752.545448] Lustre: Skipped 3 previous similar messages [ 1752.552892] Lustre: Skipped 3 previous similar messages [ 1758.295885] Lustre: server umount lustre-MDT0000 complete [ 1759.400694] LustreError: 101479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775833685 with bad export cookie 16085488490353022 [ 1759.402057] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1759.404127] LustreError: 101479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1759.510983] Lustre: server umount lustre-MDT0001 complete [ 1771.016210] Lustre: server umount lustre-OST0000 complete [ 1782.422803] Lustre: server umount lustre-OST0001 complete [ 1784.263206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1787.185118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1789.831547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1792.768863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1798.533407] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 1801.790477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1801.800449] Lustre: lustre-MDT0000: reset Object Index mappings [ 1801.987242] LustreError: 109377:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1801.992223] LustreError: 109377:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1802.013268] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1803.257717] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1805.956830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1805.963479] Lustre: lustre-MDT0001: reset Object Index mappings [ 1807.486435] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1808.465306] Lustre: 110491:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1810.528817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1811.745705] LustreError: 111034:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1811.751016] LustreError: 111034:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1811.752072] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 1812.472902] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1815.204641] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1816.423552] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 1816.424528] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 1816.425640] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 1817.159515] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1820.102217] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1821.224257] Lustre: 112337:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1841.051121] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 11:09:26 (1775833766) [ 1872.868276] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1872.868830] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1872.869515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1872.869519] Lustre: Skipped 7 previous similar messages [ 1872.872110] Lustre: Skipped 3 previous similar messages [ 1876.589443] Lustre: server umount lustre-MDT0000 complete [ 1877.804205] LustreError: 112340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775833804 with bad export cookie 16085488490399789 [ 1877.809630] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1877.983931] LustreError: 112360:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1877.989552] LustreError: 112360:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1892.319121] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1892.413752] Lustre: server umount lustre-MDT0001 complete [ 1908.191147] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1908.235856] Lustre: server umount lustre-OST0000 complete [ 1915.610483] Lustre: server umount lustre-OST0001 complete [ 1917.711901] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 1920.848184] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 1932.672960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1932.757250] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1932.771159] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1932.807207] Lustre: lustre-MDT0000: new disk, initializing [ 1932.831029] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1932.834275] Lustre: Skipped 3 previous similar messages [ 1932.841381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1934.139163] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1937.132590] Lustre: server umount lustre-MDT0000 complete [ 1940.364744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1940.485666] Lustre: Found index 0 for lustre-MDT0000, updating log [ 1940.492413] Lustre: 116548:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1940.499280] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 1941.932387] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1943.664797] Lustre: server umount lustre-MDT0000 complete [ 1946.516574] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 11:11:11 (1775833871) [ 1947.073110] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 1947.669248] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 11:11:12 (1775833872) [ 1953.320908] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 1956.721240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1958.195415] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1961.088419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1961.132352] Lustre: 118564:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1961.157737] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1961.159542] Lustre: Skipped 1 previous similar message [ 1961.198683] Lustre: lustre-MDT0001: new disk, initializing [ 1961.247542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1961.251607] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1962.553377] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1964.836951] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 1967.051399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1967.175174] Lustre: lustre-OST0000: new disk, initializing [ 1967.178190] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1968.954718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1968.959663] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1968.994132] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1969.144343] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1973.430796] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1974.593805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1974.598945] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1974.631690] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1975.453600] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 1979.991867] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1981.188049] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2100.117262] Lustre: 127314:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 2100.120622] Lustre: 127314:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 2100.122837] Lustre: 127314:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 2100.126579] Lustre: 127314:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 2100.129269] Lustre: 127314:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 2100.131677] Lustre: 127314:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 2178.632905] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 11:15:03 (1775834103) [ 2191.839971] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2191.843846] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2191.849777] Lustre: Skipped 3 previous similar messages [ 2191.852459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2191.855455] Lustre: Skipped 9 previous similar messages [ 2196.356280] Lustre: server umount lustre-MDT0000 complete [ 2197.596029] LustreError: 121703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775834124 with bad export cookie 16085488490452877 [ 2197.599502] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2197.599732] LustreError: 121703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2200.032174] LustreError: 118569:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2200.038530] LustreError: 118569:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2203.810387] Lustre: server umount lustre-MDT0001 complete [ 2211.269167] Lustre: server umount lustre-OST0000 complete [ 2218.648210] Lustre: server umount lustre-OST0001 complete [ 2220.651268] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 2223.480199] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 2234.601248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2234.689714] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2234.701713] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2234.703594] Lustre: Skipped 1 previous similar message [ 2234.736612] Lustre: lustre-MDT0000: new disk, initializing [ 2234.738848] Lustre: Skipped 1 previous similar message [ 2234.763805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2234.766450] Lustre: Skipped 5 previous similar messages [ 2234.772612] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2235.994268] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2239.983522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2240.022284] Lustre: 130303:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2240.124283] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2241.300055] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2244.266252] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2245.399970] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2247.735406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2249.849892] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2249.852879] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2249.888181] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2253.000955] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2256.134563] Lustre: lustre-MDD0000: changelog on [ 2257.023853] Lustre: lustre-MDD0001: changelog on [ 2590.322106] Lustre: lustre-MDD0001: changelog off [ 2590.969646] Lustre: 132895:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 2591.285536] Lustre: lustre-MDD0000: changelog off [ 2593.250301] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2593.250465] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2593.254291] Lustre: Skipped 5 previous similar messages [ 2593.256856] LustreError: Skipped 1 previous similar message [ 2593.257611] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2593.263675] Lustre: Skipped 9 previous similar messages [ 2599.127289] Lustre: server umount lustre-OST0000 complete [ 2606.543271] Lustre: server umount lustre-MDT0000 complete [ 2607.792233] LustreError: 130297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775834534 with bad export cookie 16085488491256393 [ 2607.795056] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2607.796969] LustreError: 130297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2607.926313] Lustre: server umount lustre-MDT0001 complete [ 2611.436567] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 2612.471206] Key type lgssc unregistered [ 2612.609304] LNet: 140256:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2612.612066] LNetError: 140256:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2612.621325] LNet: Removed LNI 192.168.203.120@tcp [ 2612.936139] Key type .llcrypt unregistered [ 2612.938211] Key type ._llcrypt unregistered [ 2619.317355] Key type ._llcrypt registered [ 2619.318533] Key type .llcrypt registered [ 2619.355498] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 2625.585765] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 2625.912164] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2625.919062] alg: No test for adler32 (adler32-zlib) [ 2626.774470] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 2626.854119] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 2628.431153] Key type lgssc registered [ 2628.810337] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2640.231274] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2640.236885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2641.323744] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2641.335356] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2641.369936] Lustre: lustre-MDT0000: new disk, initializing [ 2641.393599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2641.400412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2642.680940] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2647.036452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2647.074179] Lustre: 143395:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2647.096997] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2647.099101] Lustre: Skipped 1 previous similar message [ 2647.132427] Lustre: lustre-MDT0001: new disk, initializing [ 2647.164174] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2647.175853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2647.179475] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2648.461954] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2651.567877] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2652.716800] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2655.056015] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2655.176807] Lustre: lustre-OST0000: new disk, initializing [ 2655.179273] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2655.213995] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2657.181445] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2658.104785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2658.108844] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2658.140388] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2660.401497] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2662.666760] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_module obdecho/obdecho [ 2663.304234] LustreError: 146010:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 2663.308885] LustreError: 146010:0:(obd_config.c:843:class_setup()) setup ec failed (-22) [ 2663.941163] LustreError: 146106:0:(echo_client.c:1880:echo_md_handler()) Can not resolve the path : rc = -2 [ 2667.999859] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2668.003240] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2668.010945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2668.512204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2668.513866] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2668.518270] Lustre: Skipped 1 previous similar message [ 2673.307779] Lustre: server umount lustre-MDT0000 complete [ 2674.580362] LustreError: 145136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775834601 with bad export cookie 16326352624846063196 [ 2674.584826] LustreError: 145136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2674.586400] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2674.704591] Lustre: server umount lustre-MDT0001 complete [ 2686.233926] Lustre: server umount lustre-OST0000 complete [ 2693.277369] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 2696.658413] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 2708.474388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2708.558072] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2708.567923] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2708.603403] Lustre: lustre-MDT0000: new disk, initializing [ 2708.626710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2708.632982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2709.878785] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2713.978975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2714.016277] Lustre: 149780:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2714.037348] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2714.039742] Lustre: Skipped 1 previous similar message [ 2714.075119] Lustre: lustre-MDT0001: new disk, initializing [ 2714.106967] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2714.118681] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2714.122252] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2715.470902] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2718.659837] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2719.799400] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2722.031260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2722.148428] Lustre: lustre-OST0000: new disk, initializing [ 2722.150647] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2722.178538] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2724.074463] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2724.280237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2724.283420] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2724.314697] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2727.186751] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2728.661409] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2728.746984] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2730.115187] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2730.193230] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2734.563993] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2734.564208] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2734.566448] LustreError: Skipped 1 previous similar message [ 2734.571815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2734.575691] Lustre: Skipped 3 previous similar messages [ 2739.681322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2739.683590] Lustre: Skipped 1 previous similar message [ 2744.803596] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2744.805866] Lustre: Skipped 1 previous similar message [ 2745.311099] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2745.372177] Lustre: server umount lustre-OST0000 complete [ 2749.920836] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2749.920895] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2749.926718] Lustre: Skipped 1 previous similar message [ 2752.857719] Lustre: server umount lustre-MDT0000 complete [ 2754.101262] LustreError: 149770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775834680 with bad export cookie 16326352624846064141 [ 2754.104040] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2754.106413] LustreError: 149770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2754.217793] Lustre: server umount lustre-MDT0001 complete [ 2755.437198] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 11:24:40 (1775834680) [ 2755.988045] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 2756.636170] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 11:24:41 (1775834681) [ 2759.232046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2759.386778] LustreError: 153783:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2759.395304] LustreError: 153783:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2759.410353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2760.793894] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2763.691552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2765.287244] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2767.169832] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2768.242091] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2770.384239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2772.321318] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2774.088088] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2794.464281] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2794.467860] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2794.473548] Lustre: Skipped 1 previous similar message [ 2794.477201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2794.481376] Lustre: Skipped 2 previous similar messages [ 2797.544500] Lustre: server umount lustre-MDT0000 complete [ 2798.869472] LustreError: 153766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775834725 with bad export cookie 16326352624846065009 [ 2798.872895] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2798.873468] LustreError: 153766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2799.752529] LustreError: 153778:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2803.616046] LustreError: 153780:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2803.621593] LustreError: 153780:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2805.040393] Lustre: server umount lustre-MDT0001 complete [ 2807.321584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2811.761665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2811.863099] LustreError: 157307:0:(llog.c:1646:llog_backup()) MGC192.168.203.120@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 2811.867160] Lustre: 157307:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.203.120@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 2829.280121] LustreError: 157311:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2829.286452] LustreError: 157311:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2834.400089] LustreError: 157313:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2834.405014] LustreError: 157313:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2843.616460] LustreError: 157313:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2843.620862] LustreError: 157313:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 2861.192453] LustreError: 157312:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2861.197652] LustreError: 157312:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 2872.927289] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 2872.930870] Lustre: 157307:0:(mgc_request.c:1917:mgc_process_log()) MGC192.168.203.120@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 2893.793150] LustreError: 157313:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2893.797706] LustreError: 157313:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 21 previous similar messages [ 2913.766923] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2913.769093] Lustre: Skipped 2 previous similar messages [ 2915.114228] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2917.989780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2918.884717] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2920.343783] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2922.156983] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2922.738411] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2924.013174] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2924.179760] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2927.847379] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2932.195764] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2932.196293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2932.200957] Lustre: Skipped 4 previous similar messages [ 2932.205480] Lustre: Skipped 10 previous similar messages [ 2934.301597] Lustre: server umount lustre-MDT0000 complete [ 2935.591074] LustreError: 153765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775834862 with bad export cookie 16326352624846071449 [ 2935.593702] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2935.595703] LustreError: 153765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2935.706222] Lustre: server umount lustre-MDT0001 complete [ 2947.235252] Lustre: server umount lustre-OST0000 complete [ 2953.958792] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 2957.415737] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 2969.264038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2969.346104] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2969.356084] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2969.391301] Lustre: lustre-MDT0000: new disk, initializing [ 2969.417520] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2970.667076] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2974.723468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2974.763173] Lustre: 162122:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2974.864815] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2976.091111] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2979.066436] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2980.199414] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2982.403778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2982.541382] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2982.544154] Lustre: Skipped 3 previous similar messages [ 2983.676205] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2983.712154] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2984.390275] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 2987.445421] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2988.781338] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2988.850521] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2990.136779] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2990.208566] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2994.147132] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2994.147211] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2994.150071] LustreError: Skipped 1 previous similar message [ 2994.155117] Lustre: Skipped 1 previous similar message [ 2994.155552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2994.160297] Lustre: Skipped 1 previous similar message [ 2996.939120] Lustre: server umount lustre-OST0000 complete [ 3004.310244] Lustre: server umount lustre-MDT0000 complete [ 3004.384791] LustreError: 162916:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3004.390093] LustreError: 162916:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 18 previous similar messages [ 3005.510323] LustreError: 162115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775834932 with bad export cookie 16326352624846088249 [ 3005.513392] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3005.514925] LustreError: 162115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3005.636269] Lustre: server umount lustre-MDT0001 complete [ 3008.232908] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 11:28:53 (1775834933) [ 3010.289685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3011.680535] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3014.884126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3016.627230] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3018.301214] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3019.418431] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3021.572477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3023.408042] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3025.082113] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3039.663871] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 11:29:24 (1775834964) [ 3045.346874] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3045.347897] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3045.356940] Lustre: Skipped 4 previous similar messages [ 3055.071092] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3055.122546] Lustre: server umount lustre-OST0000 complete [ 3060.706270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3060.708179] Lustre: Skipped 7 previous similar messages [ 3062.617874] Lustre: server umount lustre-MDT0000 complete [ 3063.755058] LustreError: 166106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775834990 with bad export cookie 16326352624846089110 [ 3063.757369] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3063.758815] LustreError: 166106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3063.868392] Lustre: server umount lustre-MDT0001 complete [ 3067.325660] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 3068.320567] Key type lgssc unregistered [ 3068.457385] LNet: 169517:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3068.459978] LNetError: 169517:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3068.469274] LNet: Removed LNI 192.168.203.120@tcp [ 3068.739384] Key type .llcrypt unregistered [ 3068.740560] Key type ._llcrypt unregistered [ 3075.717933] Key type ._llcrypt registered [ 3075.719185] Key type .llcrypt registered [ 3075.752289] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 3076.073317] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3076.105361] alg: No test for adler32 (adler32-zlib) [ 3076.950225] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 3077.026862] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 3078.599105] Key type lgssc registered [ 3078.920293] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3081.368453] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3081.374781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3096.799320] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3101.919184] LustreError: 170719:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.120@tcp: failed processing log, type 4: rc = -110 [ 3107.039364] LustreError: 170719:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 3107.043861] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 3107.046137] LustreError: MGC192.168.203.120@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 3107.050307] LustreError: 170719:0:(tgt_mount.c:1819:server_start_targets()) failed to start server lustre-OST0000: -22 [ 3107.053360] LustreError: 170719:0:(tgt_mount.c:2545:server_fill_super()) Unable to start targets: -22 [ 3107.056248] LustreError: 170719:0:(tgt_mount.c:2063:server_put_super()) no obd lustre-OST0000 [ 3107.076119] Lustre: server umount lustre-OST0000 complete [ 3107.077787] LustreError: 170719:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 3109.894297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3111.054036] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3112.260521] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3115.073912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3115.191648] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3116.410794] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3118.108396] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3119.364694] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3121.665657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3121.743769] LustreError: 172804:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 3121.747640] Lustre: 172804:0:(mgc_request_server.c:743:mgc_llog_local_copy()) MGC192.168.203.120@tcp: invalid local config log lustre-OST0000: rc = -22 [ 3121.794821] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3123.668587] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3125.399084] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3130.338664] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3130.341591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3130.343212] Lustre: Skipped 1 previous similar message [ 3132.243049] Lustre: server umount lustre-MDT0000 complete [ 3133.448087] LustreError: 171199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775835060 with bad export cookie 7960946909645046376 [ 3133.450936] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3133.451820] LustreError: 171199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3133.564443] Lustre: server umount lustre-MDT0001 complete [ 3145.059792] Lustre: server umount lustre-OST0000 complete [ 3148.106694] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 3151.368914] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 3162.445733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3162.531612] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3162.544962] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3162.589548] Lustre: lustre-MDT0000: new disk, initializing [ 3162.614050] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3162.620087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3163.881133] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3167.951105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3167.990456] Lustre: 176177:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3168.014366] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3168.017348] Lustre: Skipped 1 previous similar message [ 3168.058780] Lustre: lustre-MDT0001: new disk, initializing [ 3168.090834] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3168.101624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3168.105076] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3169.391536] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3172.408513] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3173.628892] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3175.934607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3176.044038] Lustre: lustre-OST0000: new disk, initializing [ 3176.045799] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3178.052487] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3180.533968] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3180.538587] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3180.564651] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3181.038565] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3182.300499] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3182.368116] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3183.614876] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3183.683958] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3185.633619] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3185.634695] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3185.637398] Lustre: Skipped 1 previous similar message [ 3185.640615] LustreError: Skipped 1 previous similar message [ 3185.641161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3185.646608] Lustre: Skipped 3 previous similar messages [ 3186.382499] Lustre: server umount lustre-OST0000 complete [ 3188.704079] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3188.708144] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3188.715130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3190.752183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3190.752789] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3193.752885] Lustre: server umount lustre-MDT0000 complete [ 3194.976157] LustreError: 176170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775835121 with bad export cookie 7960946909645047062 [ 3194.978811] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3194.979488] LustreError: 176170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3195.097772] Lustre: server umount lustre-MDT0001 complete [ 3197.964733] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 11:32:02 (1775835122) [ 3203.884731] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 3204.963866] Key type lgssc unregistered [ 3205.102704] LNet: 180694:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3205.106219] LNetError: 180694:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3205.118438] LNet: Removed LNI 192.168.203.120@tcp [ 3205.444227] Key type .llcrypt unregistered [ 3205.445348] Key type ._llcrypt unregistered [ 3212.825430] Key type ._llcrypt registered [ 3212.826619] Key type .llcrypt registered [ 3212.858977] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 3213.142907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3213.150030] alg: No test for adler32 (adler32-zlib) [ 3214.007522] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 3214.103584] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 3215.687126] Key type lgssc registered [ 3216.022226] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3218.480510] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3218.486299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3219.639574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3220.909893] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3223.728307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3223.851503] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3225.212467] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3226.968076] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3228.134696] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3230.688863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3230.783455] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3230.784742] LustreError: 183505:0:(llog.c:1687:llog_backup()) MGC192.168.203.120@tcp: failed to backup log lustre-OST0000: rc = -28 [ 3230.787703] Lustre: 183505:0:(mgc_request_server.c:740:mgc_llog_local_copy()) MGC192.168.203.120@tcp: can't backup local config lustre-OST0000: rc = -28 [ 3230.826529] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3232.723447] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3234.539240] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3239.394308] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3239.395668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3239.398206] Lustre: Skipped 2 previous similar messages [ 3239.401052] Lustre: Skipped 1 previous similar message [ 3239.403749] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3241.367519] Lustre: server umount lustre-MDT0000 complete [ 3242.538294] LustreError: 181854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775835169 with bad export cookie 6339638759811355907 [ 3242.541075] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3242.542965] LustreError: 181854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3242.652832] Lustre: server umount lustre-MDT0001 complete [ 3254.097503] Lustre: server umount lustre-OST0000 complete [ 3257.085441] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 3260.545558] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 3272.223065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3272.311990] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3272.323127] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3272.357780] Lustre: lustre-MDT0000: new disk, initializing [ 3272.379353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3272.385248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3273.597064] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3277.689117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3277.727116] Lustre: 186882:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3277.744497] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3277.745944] Lustre: Skipped 1 previous similar message [ 3277.747391] LustreError: 187645:0:(llog.c:1687:llog_backup()) MGC192.168.203.120@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 3277.750578] LustreError: 187645:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 3277.752995] Lustre: 187645:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.203.120@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 3277.760663] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3277.762547] Lustre: Skipped 1 previous similar message [ 3277.798019] Lustre: lustre-MDT0001: new disk, initializing [ 3277.827488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3277.858846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3277.862919] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3279.124894] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3282.332036] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3283.608395] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3285.842510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3285.948135] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3285.949363] Lustre: Skipped 1 previous similar message [ 3285.951243] LustreError: 188599:0:(llog.c:1687:llog_backup()) MGC192.168.203.120@tcp: failed to backup log lustre-OST0000: rc = -28 [ 3285.955635] LustreError: 188599:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 3285.957698] Lustre: 188599:0:(mgc_request_server.c:763:mgc_llog_local_copy()) MGC192.168.203.120@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 3285.962374] Lustre: 188599:0:(mgc_request_server.c:763:mgc_llog_local_copy()) Skipped 1 previous similar message [ 3285.970804] Lustre: lustre-OST0000: new disk, initializing [ 3285.972721] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3286.001886] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3287.959031] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3290.998765] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3292.155094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3292.158975] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3292.193530] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3292.488957] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3292.577692] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3293.912923] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3293.986224] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3297.250728] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3297.252195] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3297.253457] LustreError: Skipped 1 previous similar message [ 3297.253960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3297.258257] Lustre: Skipped 1 previous similar message [ 3297.259866] Lustre: Skipped 1 previous similar message [ 3300.683590] Lustre: server umount lustre-OST0000 complete [ 3302.368660] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3302.369549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3302.373648] Lustre: Skipped 1 previous similar message [ 3302.376105] Lustre: Skipped 1 previous similar message [ 3307.487955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3307.489896] Lustre: Skipped 2 previous similar messages [ 3312.607984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3312.610238] Lustre: Skipped 1 previous similar message [ 3316.191096] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3316.253130] Lustre: server umount lustre-MDT0000 complete [ 3316.704299] LustreError: 186892:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3316.709834] LustreError: 186892:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3317.482937] LustreError: 186873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775835244 with bad export cookie 6339638759811356607 [ 3317.485941] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3317.488378] LustreError: 186873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3317.598367] Lustre: server umount lustre-MDT0001 complete [ 3320.581588] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 11:34:05 (1775835245) [ 3325.485964] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 3328.688585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3328.819508] LustreError: 191581:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3328.833483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3330.131644] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3332.866643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3334.226430] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3335.152779] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3337.314123] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3339.295274] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3342.228311] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3342.290908] Lustre: lustre-OST0001: new disk, initializing [ 3342.292938] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3343.106517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3343.109794] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3343.143168] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3344.153975] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3349.389434] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3351.907111] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3357.509301] Lustre: DEBUG MARKER: ADD OST3 [ 3359.077744] loop: module loaded [ 3359.095274] loop0: detected capacity change from 0 to 204800000 [ 3359.169020] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3359.172446] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3359.176577] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3361.172903] loop0: detected capacity change from 0 to 204800000 [ 3362.980811] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3363.041990] Lustre: lustre-OST0003: new disk, initializing [ 3363.043904] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 3363.079345] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 3363.082812] Lustre: Skipped 3 previous similar messages [ 3364.140659] Lustre: DEBUG MARKER: STOP OST3 [ 3364.303106] Lustre: *** cfs_fail_loc=2109, val=2*** [ 3364.331080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 3364.334673] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 3364.345354] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 3364.811583] Lustre: Failing over lustre-OST0003 [ 3364.854280] Lustre: server umount lustre-OST0003 complete [ 3365.130356] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3366.367199] LustreError: 195801:0:(osp_precreate.c:1280:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 3369.206477] Lustre: DEBUG MARKER: START OST3 again [ 3369.443153] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3369.446966] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3369.447878] LustreError: 193032:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3369.455525] LustreError: 193032:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3371.359519] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3371.443045] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 3372.973608] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3373.228289] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3373.402815] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3373.403356] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 3386.690523] Lustre: Failing over lustre-OST0003 [ 3386.729357] Lustre: server umount lustre-OST0003 complete [ 3386.853522] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3386.859306] Lustre: Skipped 1 previous similar message [ 3386.861641] LustreError: 193032:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3386.871185] LustreError: 193032:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3391.973157] LustreError: 193033:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3391.978321] LustreError: 193033:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3399.648155] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3399.650711] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3399.654656] Lustre: Skipped 1 previous similar message [ 3399.657184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3402.211976] LustreError: 193032:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3402.217459] LustreError: 193032:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3405.344058] Lustre: server umount lustre-MDT0000 complete [ 3406.540249] LustreError: 191562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775835333 with bad export cookie 6339638759811357475 [ 3406.543509] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3406.544535] LustreError: 191562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3412.843684] Lustre: server umount lustre-MDT0001 complete [ 3414.174568] Lustre: server umount lustre-OST0000 complete [ 3415.457919] Lustre: server umount lustre-OST0001 complete [ 3417.515202] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 3419.983143] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 3432.267819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3432.364327] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3432.377136] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3432.413326] Lustre: lustre-MDT0000: new disk, initializing [ 3432.440679] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3432.443396] Lustre: Skipped 1 previous similar message [ 3432.448695] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3433.696067] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3437.757726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3437.796986] Lustre: 200895:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3437.893735] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3439.142806] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3442.118958] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3443.355180] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3445.646469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3447.040979] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3447.088170] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3447.702498] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3450.727696] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3452.180490] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3452.264216] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3453.553323] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3453.633307] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3457.508129] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3457.508857] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3457.510018] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3457.510023] Lustre: Skipped 8 previous similar messages [ 3457.511660] Lustre: Skipped 6 previous similar messages [ 3457.523075] LustreError: Skipped 1 previous similar message [ 3460.370547] Lustre: server umount lustre-OST0000 complete [ 3467.675188] Lustre: server umount lustre-MDT0000 complete [ 3467.743985] LustreError: 200907:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3467.749648] LustreError: 200907:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 3468.904225] LustreError: 200889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775835395 with bad export cookie 6339638759811359729 [ 3468.907144] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3468.907931] LustreError: 200889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3469.024420] Lustre: server umount lustre-MDT0001 complete [ 3470.223936] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 11:36:35 (1775835395) [ 3475.276782] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 3478.556184] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 3489.924116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3490.013700] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3490.026220] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3490.028210] Lustre: Skipped 3 previous similar messages [ 3490.064767] Lustre: lustre-MDT0000: new disk, initializing [ 3490.066212] Lustre: Skipped 2 previous similar messages [ 3490.095800] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3490.098927] Lustre: Skipped 2 previous similar messages [ 3491.421495] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3495.649248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3495.689552] Lustre: 207056:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3495.797533] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3497.088151] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3500.247282] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3501.480940] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3503.851454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3505.886481] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3505.894030] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3508.962138] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3510.290705] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3510.368222] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3511.680185] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3511.751928] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3516.385201] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3516.386748] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3516.391062] LustreError: Skipped 1 previous similar message [ 3516.394940] Lustre: Skipped 2 previous similar messages [ 3516.395638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3516.402710] Lustre: Skipped 4 previous similar messages [ 3518.479504] Lustre: server umount lustre-OST0000 complete [ 3525.851534] Lustre: server umount lustre-MDT0000 complete [ 3526.624321] LustreError: 207067:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3526.631642] LustreError: 207067:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3527.072181] LustreError: 207048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775835453 with bad export cookie 6339638759811360611 [ 3527.074601] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3527.076870] LustreError: 207048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3527.198766] Lustre: server umount lustre-MDT0001 complete [ 3529.960297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3531.429751] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3534.201995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3535.600395] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3537.348820] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3538.557340] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3540.804042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3542.976767] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3544.716102] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3566.559689] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3566.562533] LustreError: Skipped 1 previous similar message [ 3572.690141] Lustre: server umount lustre-OST0000 complete [ 3580.055308] Lustre: server umount lustre-MDT0000 complete [ 3581.299815] LustreError: 210846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775835507 with bad export cookie 6339638759811361493 [ 3581.302200] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3581.305415] LustreError: 210846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3581.436215] Lustre: server umount lustre-MDT0001 complete [ 3678.387685] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 3679.368262] Key type lgssc unregistered [ 3679.495353] LNet: 214197:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3679.497782] LNetError: 214197:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3679.505276] LNet: Removed LNI 192.168.203.120@tcp [ 3679.804104] Key type .llcrypt unregistered [ 3679.805287] Key type ._llcrypt unregistered [ 3688.041086] Key type ._llcrypt registered [ 3688.042281] Key type .llcrypt registered [ 3688.087819] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 3694.457706] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 3694.836099] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3694.906535] alg: No test for adler32 (adler32-zlib) [ 3695.777544] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 3695.862691] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 3697.447157] Key type lgssc registered [ 3697.833789] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3709.177836] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3709.184306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3710.302175] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3710.331216] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3710.372436] Lustre: lustre-MDT0000: new disk, initializing [ 3710.400751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3710.410092] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3711.745278] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3715.851169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3715.890217] Lustre: 217493:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3715.913253] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3715.915286] Lustre: Skipped 1 previous similar message [ 3715.952835] Lustre: lustre-MDT0001: new disk, initializing [ 3715.983667] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3715.994950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3715.998096] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3717.282648] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3720.349235] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3721.518271] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3723.759177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3723.857778] Lustre: lustre-OST0000: new disk, initializing [ 3723.859573] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3723.888221] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3725.370329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3725.373556] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3725.402837] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3725.730100] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3728.785348] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3730.139334] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3730.206919] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3731.524256] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3731.598523] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3735.522445] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3735.522452] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3735.523172] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3735.526506] Lustre: Skipped 1 previous similar message [ 3735.533122] Lustre: Skipped 1 previous similar message [ 3738.326854] Lustre: server umount lustre-OST0000 complete [ 3740.640823] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3740.641588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3740.645074] Lustre: Skipped 1 previous similar message [ 3740.648782] Lustre: Skipped 1 previous similar message [ 3745.754791] Lustre: server umount lustre-MDT0000 complete [ 3747.013272] LustreError: 217485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775835673 with bad export cookie 14547917860436051056 [ 3747.017457] LustreError: 217485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3747.017582] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3747.140852] Lustre: server umount lustre-MDT0001 complete [ 3753.308976] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 3756.954680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3757.095874] LustreError: 221997:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3757.103167] LustreError: 221997:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3757.111521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3758.422684] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3761.275499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3761.401929] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3762.749248] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3763.751518] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3766.048234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3768.185451] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3771.100516] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3771.164652] Lustre: lustre-OST0001: new disk, initializing [ 3771.166520] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3771.202513] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3771.204966] Lustre: Skipped 1 previous similar message [ 3773.217230] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3778.737315] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3779.587371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3779.590537] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3779.626790] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3784.489080] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3793.285734] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 11:41:58 (1775835718) [ 3794.913767] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3794.917122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3794.918811] Lustre: Skipped 2 previous similar messages [ 3794.921470] Lustre: Skipped 1 previous similar message [ 3800.033081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3800.035107] Lustre: Skipped 3 previous similar messages [ 3800.095377] Lustre: server umount lustre-MDT0000 complete [ 3801.293245] LustreError: 223095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775835727 with bad export cookie 14547917860436051924 [ 3801.296738] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3801.297203] LustreError: 223095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3801.412280] Lustre: server umount lustre-MDT0001 complete [ 3813.380735] Lustre: server umount lustre-OST0000 complete [ 3824.800140] Lustre: server umount lustre-OST0001 complete [ 3826.704077] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 3829.689334] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 3841.049656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3841.157250] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3841.170187] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3841.209037] Lustre: lustre-MDT0000: new disk, initializing [ 3841.233758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3841.241845] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3842.568503] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3846.658560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3846.700770] Lustre: 228142:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3846.817280] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3848.171424] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3851.229667] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3852.395280] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3854.994763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3855.122674] Lustre: lustre-OST0000: new disk, initializing [ 3855.124607] Lustre: Skipped 1 previous similar message [ 3855.126831] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3855.129331] Lustre: Skipped 2 previous similar messages [ 3857.019859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3857.023379] Lustre: Skipped 1 previous similar message [ 3857.025204] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3857.059473] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3857.321695] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3860.444040] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3861.926816] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3862.012344] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3863.380215] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3863.463730] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3867.618768] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3867.618924] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3867.619585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3867.621511] LustreError: Skipped 2 previous similar messages [ 3867.626206] Lustre: Skipped 2 previous similar messages [ 3877.857852] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3877.861582] Lustre: Skipped 4 previous similar messages [ 3878.367141] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3878.425986] Lustre: server umount lustre-OST0000 complete [ 3882.978293] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3882.978881] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3882.983032] Lustre: Skipped 1 previous similar message [ 3885.790745] Lustre: server umount lustre-MDT0000 complete [ 3886.999548] LustreError: 229886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775835813 with bad export cookie 14547917860436054052 [ 3887.002845] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3887.004279] LustreError: 229886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3887.149868] Lustre: server umount lustre-MDT0001 complete [ 3889.976304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3890.148527] LustreError: 231951:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3890.173303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3890.175714] Lustre: Skipped 2 previous similar messages [ 3891.521304] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3894.328418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3895.793853] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3897.511453] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3898.651985] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3900.885660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3902.877729] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 3904.647207] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4052.960072] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4052.964066] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4052.969676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4052.971775] Lustre: Skipped 2 previous similar messages [ 4063.199188] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4063.261284] Lustre: server umount lustre-OST0000 complete [ 4068.831907] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4068.834429] LustreError: Skipped 1 previous similar message [ 4069.345061] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4069.349365] Lustre: Skipped 2 previous similar messages [ 4070.681483] Lustre: server umount lustre-MDT0000 complete [ 4071.940680] LustreError: 231932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775835998 with bad export cookie 14547917860436054934 [ 4071.945223] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4071.945631] LustreError: 231932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4072.064215] Lustre: server umount lustre-MDT0001 complete [ 4170.179267] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 4171.163542] Key type lgssc unregistered [ 4171.290424] LNet: 235205:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4171.293235] LNetError: 235205:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4171.299459] LNet: Removed LNI 192.168.203.120@tcp [ 4171.643147] Key type .llcrypt unregistered [ 4171.644302] Key type ._llcrypt unregistered [ 4179.729869] Key type ._llcrypt registered [ 4179.731087] Key type .llcrypt registered [ 4179.766599] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 4186.014285] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 4186.463931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4186.486992] alg: No test for adler32 (adler32-zlib) [ 4187.359830] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 4187.449705] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 4189.039182] Key type lgssc registered [ 4189.441281] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4204.764653] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 4208.438726] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4208.450371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4209.559760] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4209.572104] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4209.613155] Lustre: lustre-MDT0000: new disk, initializing [ 4209.638565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4209.645588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4210.916425] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4215.076659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4215.117666] Lustre: 239193:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4215.146117] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4215.148855] Lustre: Skipped 1 previous similar message [ 4215.189321] Lustre: lustre-MDT0001: new disk, initializing [ 4215.221071] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4215.231381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4215.234521] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4216.493993] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4218.753862] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4220.817087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4220.928686] Lustre: lustre-OST0000: new disk, initializing [ 4220.931697] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4220.962587] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4222.829667] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4224.696019] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4224.699581] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4224.747524] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4226.807203] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4226.875421] Lustre: lustre-OST0001: new disk, initializing [ 4226.877274] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4226.909404] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4228.681203] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4232.219364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4232.226531] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4232.273305] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4234.368941] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4240.213056] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4251.104068] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4251.107294] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4251.112702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4252.641867] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4252.643628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4252.646021] Lustre: Skipped 2 previous similar messages [ 4252.648433] Lustre: Skipped 2 previous similar messages [ 4254.240229] Lustre: server umount lustre-MDT0000 complete [ 4255.452421] LustreError: 242398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775836182 with bad export cookie 10976683672195684858 [ 4255.455684] LustreError: 242398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4255.456063] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4255.581214] Lustre: server umount lustre-MDT0001 complete [ 4266.960923] Lustre: server umount lustre-OST0000 complete [ 4278.491300] Lustre: server umount lustre-OST0001 complete [ 4283.166209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4286.415893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4289.362271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4292.584445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4298.861068] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 4302.476187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4302.483161] Lustre: lustre-MDT0000: reset Object Index mappings [ 4302.654326] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4303.943390] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4306.798438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4306.805506] Lustre: lustre-MDT0001: reset Object Index mappings [ 4308.250435] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4309.305521] Lustre: 247722:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4311.699754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4311.845093] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4311.847095] Lustre: Skipped 1 previous similar message [ 4312.869636] LustreError: 248076:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4312.872906] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4312.874459] LustreError: 248076:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4313.777417] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4316.963214] LustreError: 248077:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4316.971443] LustreError: 248077:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4317.053153] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4318.589464] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 4319.238195] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4322.621826] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4324.066691] Lustre: 249565:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4332.515232] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4332.515282] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4332.516719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4332.516728] Lustre: Skipped 1 previous similar message [ 4332.518761] Lustre: Skipped 2 previous similar messages [ 4336.483803] Lustre: server umount lustre-MDT0000 complete [ 4337.635650] LustreError: 247329:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4337.641920] LustreError: 247329:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4337.728490] LustreError: 246595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775836264 with bad export cookie 10976683672195691634 [ 4337.732326] LustreError: 246595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4337.733086] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4337.865661] Lustre: server umount lustre-MDT0001 complete [ 4348.949467] Lustre: server umount lustre-OST0000 complete [ 4359.540595] Lustre: server umount lustre-OST0001 complete [ 4366.251868] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 11:51:31 (1775836291) [ 4371.656082] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 4375.231365] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 4386.693393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4386.789206] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4386.803390] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4386.846720] Lustre: lustre-MDT0000: new disk, initializing [ 4386.873685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4386.876733] Lustre: Skipped 1 previous similar message [ 4386.883816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4388.228658] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4392.447975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4392.491165] Lustre: 253778:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4392.591063] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4393.893111] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4396.878062] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4398.038975] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4400.194237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4400.307235] Lustre: lustre-OST0000: new disk, initializing [ 4400.309272] Lustre: Skipped 1 previous similar message [ 4400.311792] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4400.314371] Lustre: Skipped 2 previous similar messages [ 4401.397339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4401.400190] Lustre: Skipped 1 previous similar message [ 4401.401973] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4401.430544] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4402.133630] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4405.111259] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4406.440666] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4406.512165] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4407.772740] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4407.845195] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4411.874665] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4411.874882] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4411.875244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4411.875249] Lustre: Skipped 3 previous similar messages [ 4411.878064] Lustre: Skipped 1 previous similar message [ 4414.548098] Lustre: server umount lustre-OST0000 complete [ 4416.992724] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4416.993621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4416.996781] Lustre: Skipped 1 previous similar message [ 4417.001537] Lustre: Skipped 2 previous similar messages [ 4427.232230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4427.234338] Lustre: Skipped 3 previous similar messages [ 4430.303123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4430.362840] Lustre: server umount lustre-MDT0000 complete [ 4431.557325] LustreError: 253771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775836358 with bad export cookie 10976683672195693727 [ 4431.560589] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4431.560995] LustreError: 253771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4431.674539] Lustre: server umount lustre-MDT0001 complete [ 4437.236664] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 4440.643267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4440.795717] LustreError: 258287:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4440.815202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4440.817230] Lustre: Skipped 2 previous similar messages [ 4442.097064] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4444.881324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4446.320440] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4447.255397] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4449.379576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4451.276173] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4454.009711] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4454.071300] Lustre: lustre-OST0001: new disk, initializing [ 4454.073350] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4455.961445] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4456.071260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4456.074546] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4456.101978] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4460.473293] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4461.705548] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4468.504076] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 4468.509662] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 4470.752178] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4470.754649] LustreError: Skipped 1 previous similar message [ 4470.756025] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4470.760405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4476.329060] Lustre: server umount lustre-MDT0000 complete [ 4476.898836] LustreError: 258997:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4476.904071] LustreError: 258997:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4477.565312] LustreError: 258267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775836404 with bad export cookie 10976683672195694609 [ 4477.568318] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4477.568983] LustreError: 258267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4477.685876] Lustre: server umount lustre-MDT0001 complete [ 4489.220107] Lustre: server umount lustre-OST0000 complete [ 4499.607675] Lustre: server umount lustre-OST0001 complete [ 4504.832556] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 4508.173663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4508.321929] LustreError: 263561:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4508.340499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4508.342880] Lustre: Skipped 3 previous similar messages [ 4509.613035] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4512.532755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4514.053358] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4515.030219] Lustre: 264669:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4517.162614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4518.318219] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 4519.217306] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4521.997950] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4523.127930] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 4523.986708] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4526.948558] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4528.097441] Lustre: 266517:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4534.719051] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 4534.723907] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 4537.569160] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 11:54:22 (1775836462) [ 4549.093378] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4549.096403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4549.097024] Lustre: Skipped 6 previous similar messages [ 4549.099029] Lustre: Skipped 6 previous similar messages [ 4555.161780] Lustre: server umount lustre-MDT0000 complete [ 4556.306100] LustreError: 264285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775836482 with bad export cookie 10976683672195700916 [ 4556.309351] LustreError: 264285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4556.309452] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4559.327927] LustreError: 265205:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4559.333935] LustreError: 265205:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4562.474654] Lustre: server umount lustre-MDT0001 complete [ 4569.723421] Lustre: server umount lustre-OST0000 complete [ 4576.980622] Lustre: server umount lustre-OST0001 complete [ 4578.874409] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 4582.021732] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 4596.302293] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 4599.536142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4599.624178] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4599.635253] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4599.677490] Lustre: lustre-MDT0000: new disk, initializing [ 4599.705726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4600.919354] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4604.977014] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4605.017469] Lustre: 270695:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4605.119757] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4606.380194] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4610.615483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4611.144477] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4612.544370] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4616.534145] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4618.488477] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4618.562219] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4622.920136] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4632.879113] Lustre: DEBUG MARKER: == conf-sanity test 160: MGC updates failnodes from all participants ========================================================== 11:55:57 (1775836557) [ 4634.081102] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4634.084471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4634.084806] Lustre: Skipped 5 previous similar messages [ 4634.086990] Lustre: Skipped 10 previous similar messages [ 4639.703250] Lustre: server umount lustre-MDT0000 complete [ 4640.889215] LustreError: 270689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775836567 with bad export cookie 10976683672195708798 [ 4640.893453] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4640.893558] LustreError: 270689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4641.011066] Lustre: server umount lustre-MDT0001 complete [ 4651.528642] Lustre: server umount lustre-OST0000 complete [ 4662.996572] Lustre: server umount lustre-OST0001 complete [ 4668.569074] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 4671.746113] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 4683.036519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4683.136799] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4683.141931] Lustre: Skipped 2 previous similar messages [ 4683.154322] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4683.156268] Lustre: Skipped 4 previous similar messages [ 4683.192910] Lustre: lustre-MDT0000: new disk, initializing [ 4683.194233] Lustre: Skipped 3 previous similar messages [ 4683.220345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4683.223088] Lustre: Skipped 7 previous similar messages [ 4683.228803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4683.231389] Lustre: Skipped 3 previous similar messages [ 4684.478492] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4688.446582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4688.483746] Lustre: 277723:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4688.578327] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4688.580964] Lustre: Skipped 2 previous similar messages [ 4689.791538] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4692.798325] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4693.976239] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4696.162046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4698.105829] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4698.205100] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4701.112167] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4701.782825] Lustre: Mounted lustre-client [ 4702.078349] LustreError: 280219:0:(lov_obd.c:786:lov_cleanup()) lustre-clilov-ffff99584777a000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 4702.104768] Lustre: Unmounted lustre-client [ 4706.786376] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4706.790327] Lustre: Skipped 1 previous similar message [ 4707.024071] Lustre: server umount lustre-OST0000 complete [ 4709.343975] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4714.394249] Lustre: server umount lustre-MDT0000 complete [ 4715.562330] LustreError: 278501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775836642 with bad export cookie 10976683672195710324 [ 4715.565574] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4715.565661] LustreError: 278501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4715.678072] Lustre: server umount lustre-MDT0001 complete [ 4718.914915] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 4719.855562] Key type lgssc unregistered [ 4753.759126] LNet: 1 local NIs in recovery (showing 1): 192.168.203.120@tcp [ 4753.761553] LNet: 281497:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4753.764752] LNetError: 281497:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4830.565415] LNet: Removed LNI 192.168.203.120@tcp [ 4830.871103] Key type .llcrypt unregistered [ 4830.872220] Key type ._llcrypt unregistered [ 4836.704672] Key type ._llcrypt registered [ 4836.705971] Key type .llcrypt registered [ 4836.739547] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 4843.195541] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 4843.590865] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4843.630186] alg: No test for adler32 (adler32-zlib) [ 4844.474548] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 4844.552072] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 4846.127125] Key type lgssc registered [ 4846.487151] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4856.049966] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 11:59:41 (1775836781) [ 4858.227664] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4858.233385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4859.332828] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4859.344828] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4859.385919] Lustre: lustre-MDT0000: new disk, initializing [ 4859.411768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4859.419282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4860.743379] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4864.875146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4864.916545] Lustre: 284657:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4864.946067] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4864.946804] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4864.950779] Lustre: Skipped 1 previous similar message [ 4864.986274] Lustre: lustre-MDT0001: new disk, initializing [ 4865.017955] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4865.029214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4865.032239] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4866.244283] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4869.268416] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4870.431920] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4872.803100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4872.933984] Lustre: lustre-OST0000: new disk, initializing [ 4872.937847] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4872.977589] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4874.864278] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4877.872025] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4880.887466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4880.892438] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4880.924408] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4891.104771] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4891.106467] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4891.108302] LustreError: Skipped 1 previous similar message [ 4891.109031] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4891.115084] Lustre: Skipped 1 previous similar message [ 4892.818576] Lustre: server umount lustre-OST0000 complete [ 4895.712026] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4895.716232] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4895.723453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4895.726686] Lustre: Skipped 1 previous similar message [ 4900.182755] Lustre: server umount lustre-MDT0000 complete [ 4901.339388] LustreError: 284650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775836827 with bad export cookie 8931813205507203989 [ 4901.342218] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4901.345111] LustreError: 284650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4901.456868] Lustre: server umount lustre-MDT0001 complete [ 4905.059756] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 4906.091615] Key type lgssc unregistered [ 4906.218394] LNet: 288031:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4906.220763] LNetError: 288031:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4906.231326] LNet: Removed LNI 192.168.203.120@tcp [ 4906.535300] Key type .llcrypt unregistered [ 4906.536855] Key type ._llcrypt unregistered [ 4915.514974] Key type ._llcrypt registered [ 4915.516080] Key type .llcrypt registered [ 4915.546264] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 4915.845246] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4915.888169] alg: No test for adler32 (adler32-zlib) [ 4916.739855] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 4916.821727] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 4918.399113] Key type lgssc registered [ 4918.764085] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4921.303582] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4921.308950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4922.470708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4923.747018] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4926.521449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4926.659610] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4927.940443] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4929.754214] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4930.890777] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4933.043033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4933.089132] LustreError: 290998:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4933.094299] LustreError: 290998:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4933.156700] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4934.911498] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4936.563247] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4945.451488] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4945.959569] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4947.393582] Lustre: server umount lustre-MDT0000 complete [ 4947.425384] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4947.426278] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4947.426864] LustreError: 291565:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4947.429054] Lustre: Skipped 1 previous similar message [ 4948.571165] LustreError: 289387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775836875 with bad export cookie 12592018904345835446 [ 4948.574168] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4948.574855] LustreError: 289387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4952.543874] LustreError: 289401:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4952.544136] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4952.548438] LustreError: 289401:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4952.551978] Lustre: Skipped 1 previous similar message [ 4952.556579] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4953.231648] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.20@tcp (stopping) [ 4954.727205] Lustre: server umount lustre-MDT0001 complete [ 4958.349596] Lustre: lustre-OST0000: Not available for connect from 192.168.203.20@tcp (stopping) [ 4958.351682] Lustre: Skipped 1 previous similar message [ 4962.065346] Lustre: server umount lustre-OST0000 complete [ 4964.876808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4966.203228] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4967.094803] Lustre: 293144:0:(mgs_llog.c:1709:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.120@tcp [ 4967.753860] Lustre: server umount MGS complete [ 4970.536611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4970.696171] LustreError: 293715:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4970.703737] LustreError: 293715:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4970.717636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4971.960372] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4974.590305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4975.933542] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4978.573724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4978.677127] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4978.679092] Lustre: Skipped 1 previous similar message [ 4980.471458] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 4982.056792] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4999.041652] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 16 sec [ 5000.198100] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5000.723039] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5003.849919] Lustre: server umount lustre-OST0000 complete [ 5005.280333] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5005.281088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5005.286325] Lustre: Skipped 1 previous similar message [ 5010.400068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5010.402188] Lustre: Skipped 2 previous similar messages [ 5014.518660] Lustre: server umount lustre-MDT0000 complete [ 5015.520346] LustreError: 293716:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5015.526238] LustreError: 293716:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5015.721167] LustreError: 293696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775836942 with bad export cookie 12592018904345836272 [ 5015.725646] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5015.726269] LustreError: 293696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5026.320343] Lustre: server umount lustre-MDT0001 complete [ 5029.686234] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 5030.639975] Key type lgssc unregistered [ 5030.760404] LNet: 296918:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5030.763406] LNetError: 296918:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5030.769475] LNet: Removed LNI 192.168.203.120@tcp [ 5031.059109] Key type .llcrypt unregistered [ 5031.060244] Key type ._llcrypt unregistered [ 5036.685034] Key type ._llcrypt registered [ 5036.686145] Key type .llcrypt registered [ 5036.721828] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 5042.844644] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 5043.132855] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5043.151885] alg: No test for adler32 (adler32-zlib) [ 5043.993394] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 5044.068919] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 5045.647109] Key type lgssc registered [ 5045.986608] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5057.023362] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5057.029988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5058.128996] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5058.142250] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5058.179332] Lustre: lustre-MDT0000: new disk, initializing [ 5058.205417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5058.211702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5059.551223] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 5063.559947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5063.603794] Lustre: 300014:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5063.628307] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5063.630314] Lustre: Skipped 1 previous similar message [ 5063.669750] Lustre: lustre-MDT0001: new disk, initializing [ 5063.707836] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5063.720491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5063.724442] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5065.010306] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 5068.008621] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5069.153151] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5071.224755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5071.323054] Lustre: lustre-OST0000: new disk, initializing [ 5071.324950] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5071.352629] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5073.008300] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 5075.871720] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5075.958467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5075.961640] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5075.991027] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5077.109750] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5077.173526] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5078.337486] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5078.398262] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5081.058562] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5081.059092] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5081.060747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5081.060792] LustreError: Skipped 1 previous similar message [ 5081.067077] Lustre: Skipped 1 previous similar message [ 5085.075791] Lustre: server umount lustre-OST0000 complete [ 5089.248092] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5089.250457] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5089.254805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5089.256389] Lustre: Skipped 1 previous similar message [ 5091.296985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5091.297470] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5092.377464] Lustre: server umount lustre-MDT0000 complete [ 5093.575438] LustreError: 300005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775837020 with bad export cookie 3793297301796730494 [ 5093.578756] LustreError: 300005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5093.579332] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5093.707615] Lustre: server umount lustre-MDT0001 complete [ 5096.371128] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 12:03:41 (1775837021) [ 5101.373338] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 5104.728695] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 5116.542434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5116.639171] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5116.652142] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5116.692116] Lustre: lustre-MDT0000: new disk, initializing [ 5116.717545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5116.723490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5117.985761] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 5122.010383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5122.051650] Lustre: 306320:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5122.078279] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5122.079894] Lustre: Skipped 1 previous similar message [ 5122.115189] Lustre: lustre-MDT0001: new disk, initializing [ 5122.148719] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5122.163111] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5122.166937] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5123.425597] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 5126.446755] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5127.629954] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5129.661676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5131.389109] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 5134.315131] Lustre: DEBUG MARKER: oleg320-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5134.327745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5134.333160] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5134.361590] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5144.549553] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5144.550427] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5144.551268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5144.551275] Lustre: Skipped 1 previous similar message [ 5144.551627] LustreError: Skipped 1 previous similar message [ 5144.558771] Lustre: Skipped 1 previous similar message [ 5145.297120] Lustre: server umount lustre-OST0000 complete [ 5147.615921] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5149.664234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5149.664588] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5149.665994] Lustre: Skipped 2 previous similar messages [ 5149.669354] Lustre: Skipped 1 previous similar message [ 5152.667304] Lustre: server umount lustre-MDT0000 complete [ 5153.844325] LustreError: 306311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775837080 with bad export cookie 3793297301796731362 [ 5153.846960] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5153.847928] LustreError: 306311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5153.964482] Lustre: server umount lustre-MDT0001 complete [ 5157.360235] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 5158.335295] Key type lgssc unregistered [ 5158.454428] LNet: 310074:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5158.456805] LNetError: 310074:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5158.468392] LNet: Removed LNI 192.168.203.120@tcp [ 5158.755515] Key type .llcrypt unregistered [ 5158.756964] Key type ._llcrypt unregistered [ 5188.742796] Key type ._llcrypt registered [ 5188.743985] Key type .llcrypt registered [ 5188.775701] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 5188.863391] Key type .llcrypt unregistered [ 5188.864638] Key type ._llcrypt unregistered [ 5215.011930] Key type ._llcrypt registered [ 5215.013009] Key type .llcrypt registered [ 5215.055425] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 5215.160405] Key type .llcrypt unregistered [ 5215.161412] Key type ._llcrypt unregistered [ 5226.139714] Key type ._llcrypt registered [ 5226.140978] Key type .llcrypt registered [ 5226.182844] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 5226.287271] Key type .llcrypt unregistered [ 5226.288347] Key type ._llcrypt unregistered [ 5236.694798] Key type ._llcrypt registered [ 5236.695889] Key type .llcrypt registered [ 5236.731693] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 5236.817371] Key type .llcrypt unregistered [ 5236.818961] Key type ._llcrypt unregistered [ 5263.799405] Key type ._llcrypt registered [ 5263.800632] Key type .llcrypt registered [ 5263.839287] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 5263.939438] Key type .llcrypt unregistered [ 5263.941034] Key type ._llcrypt unregistered [ 5274.444602] Key type ._llcrypt registered [ 5274.446023] Key type .llcrypt registered [ 5274.484702] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 5274.587446] Key type .llcrypt unregistered [ 5274.588504] Key type ._llcrypt unregistered [ 5285.201731] Key type ._llcrypt registered [ 5285.202869] Key type .llcrypt registered [ 5285.249052] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 5285.361224] Key type .llcrypt unregistered [ 5285.363129] Key type ._llcrypt unregistered [ 5296.078068] Key type ._llcrypt registered [ 5296.079246] Key type .llcrypt registered [ 5296.123425] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 5296.230412] Key type .llcrypt unregistered [ 5296.232199] Key type ._llcrypt unregistered [ 5306.652866] Key type ._llcrypt registered [ 5306.653842] Key type .llcrypt registered [ 5306.685019] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 5306.777176] Key type .llcrypt unregistered [ 5306.778240] Key type ._llcrypt unregistered [ 5317.667896] Key type ._llcrypt registered [ 5317.668958] Key type .llcrypt registered [ 5317.701780] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 5317.797427] Key type .llcrypt unregistered [ 5317.798572] Key type ._llcrypt unregistered [ 5341.274580] Key type ._llcrypt registered [ 5341.275745] Key type .llcrypt registered [ 5341.330918] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 5341.466171] Key type .llcrypt unregistered [ 5341.467323] Key type ._llcrypt unregistered [ 5353.296372] Key type ._llcrypt registered [ 5353.297563] Key type .llcrypt registered [ 5353.333458] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 5353.431237] Key type .llcrypt unregistered [ 5353.432493] Key type ._llcrypt unregistered [ 5363.922987] Key type ._llcrypt registered [ 5363.924694] Key type .llcrypt registered [ 5363.962757] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 5364.062228] Key type .llcrypt unregistered [ 5364.063969] Key type ._llcrypt unregistered [ 5383.277454] Key type ._llcrypt registered [ 5383.278579] Key type .llcrypt registered [ 5383.316581] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 5389.169521] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 5389.429220] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5389.451992] alg: No test for adler32 (adler32-zlib) [ 5390.307442] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 5390.387858] LNet: Added LNI 192.168.203.120@tcp [8/256/0/180] [ 5391.967158] Key type lgssc registered [ 5392.332401] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5403.120059] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5403.125648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5404.223838] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 5404.237507] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 5404.284222] Lustre: tst250fs-MDT0000: new disk, initializing [ 5404.310168] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5404.316192] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5405.626287] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 5409.726083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5409.768449] Lustre: 329707:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5409.794862] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 5409.796702] Lustre: Skipped 1 previous similar message [ 5409.832622] Lustre: tst250fs-MDT0001: new disk, initializing [ 5409.870588] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5409.882653] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5409.887968] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5411.208320] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 5413.465125] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5415.576040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5415.690968] Lustre: tst250fs-OST0000: new disk, initializing [ 5415.693914] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 5415.724136] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5417.547394] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 5419.282391] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5419.287702] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5419.356816] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5422.913475] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5423.032120] Lustre: tst250fs-OST0001: new disk, initializing [ 5423.035755] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 5423.093875] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5425.388069] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_default_debug -1 all [ 5428.120970] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 5428.841828] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5428.847210] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5428.901378] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5439.460213] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5439.462905] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 5439.464804] Lustre: Skipped 1 previous similar message [ 5439.467773] Lustre: Skipped 3 previous similar messages [ 5444.577105] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 5445.298625] Lustre: server umount tst250fs-MDT0000 complete [ 5446.753238] LustreError: 329698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775837373 with bad export cookie 13357504975389975465 [ 5446.756863] LustreError: 329698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5446.757066] LustreError: MGC192.168.203.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5446.883667] Lustre: server umount tst250fs-MDT0001 complete [ 5457.924937] Lustre: server umount tst250fs-OST0000 complete [ 5469.341162] Lustre: server umount tst250fs-OST0001 complete [ 5474.707065] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 5477.275941] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 5486.823975] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 12:10:11 (1775837411) [ 5487.304362] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 5492.342326] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing set_hostid [ 5496.308069] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing load_modules_local [ 5507.010962] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 5186 sec ========== 12:10:31 (1775837431) [ 5507.709943] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:10:32 (1775837432) === [ 5508.836789] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:10:33 (1775837433) === [ 5521.977742] Lustre: DEBUG MARKER: oleg320-server.virtnet: executing unload_modules_local [ 5522.960120] Key type lgssc unregistered [ 5523.085371] LNet: 341188:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5523.088941] LNetError: 341188:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5523.098363] LNet: Removed LNI 192.168.203.120@tcp [ 5523.443097] Key type .llcrypt unregistered [ 5523.444282] Key type ._llcrypt unregistered