[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 439602439 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001075] APIC: Switch to symmetric I/O mode setup [ 0.002440] x2apic enabled [ 0.003012] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008030] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009010] pid_max: default: 32768 minimum: 301 [ 0.010129] LSM: Security Framework initializing [ 0.011044] Yama: becoming mindful. [ 0.012029] SELinux: Initializing. [ 0.013042] *** VALIDATE selinux *** [ 0.020291] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025157] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026168] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027135] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028127] *** VALIDATE tmpfs *** [ 0.030005] *** VALIDATE proc *** [ 0.030769] *** VALIDATE cgroup *** [ 0.031011] *** VALIDATE cgroup2 *** [ 0.032333] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034179] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036046] Spectre V2 : User space: Vulnerable [ 0.037007] Speculative Store Bypass: Vulnerable [ 0.040585] debug: unmapping init [mem 0xffffffff9e259000-0xffffffff9e260fff] [ 0.043229] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044826] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045031] ... version: 2 [ 0.046011] ... bit width: 48 [ 0.047012] ... generic registers: 4 [ 0.048011] ... value mask: 0000ffffffffffff [ 0.049011] ... max period: 00007fffffffffff [ 0.050017] ... fixed-purpose events: 3 [ 0.051011] ... event mask: 000000070000000f [ 0.052352] rcu: Hierarchical SRCU implementation. [ 0.054616] smp: Bringing up secondary CPUs ... [ 0.055474] x86: Booting SMP configuration: [ 0.056016] .... node #0, CPUs: #1 #2 #3 [ 0.063816] smp: Brought up 1 node, 4 CPUs [ 0.064895] smpboot: Max logical packages: 1 [ 0.065010] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.135057] node 0 deferred pages initialised in 68ms [ 0.141381] devtmpfs: initialized [ 0.145170] x86/mm: Memory block size: 128MB [ 0.152000] gcov: version magic: 0x41383552 [ 0.156680] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.157107] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.158331] pinctrl core: initialized pinctrl subsystem [ 0.159305] [ 0.160000] ************************************************************* [ 0.163019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.166012] ** ** [ 0.168015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.171018] ** ** [ 0.174014] ** This means that this kernel is built to expose internal ** [ 0.176010] ** IOMMU data structures, which may compromise security on ** [ 0.178012] ** your system. ** [ 0.181014] ** ** [ 0.183012] ** If you see this message and you are not debugging the ** [ 0.186023] ** kernel, report this immediately to your vendor! ** [ 0.189270] ** ** [ 0.192021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.195013] ************************************************************* [ 0.198000] NET: Registered protocol family 16 [ 0.200799] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.204064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.207132] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.216042] cpuidle: using governor menu [ 0.219029] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.222958] PCI: Using configuration type 1 for base access [ 0.226217] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.240567] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.244853] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.255003] cryptd: max_cpu_qlen set to 1000 [ 0.264892] ACPI: Added _OSI(Module Device) [ 0.267017] ACPI: Added _OSI(Processor Device) [ 0.269025] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.272012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.280118] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.293092] ACPI: Interpreter enabled [ 0.296476] ACPI: PM: (supports S0 S3 S4 S5) [ 0.301041] ACPI: Using IOAPIC for interrupt routing [ 0.305363] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.313915] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.334000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.338096] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.343032] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.347137] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.354126] acpiphp: Slot [2] registered [ 0.356308] acpiphp: Slot [5] registered [ 0.357148] acpiphp: Slot [6] registered [ 0.358185] acpiphp: Slot [7] registered [ 0.359000] acpiphp: Slot [8] registered [ 0.359000] acpiphp: Slot [9] registered [ 0.359225] acpiphp: Slot [10] registered [ 0.360000] acpiphp: Slot [3] registered [ 0.362223] acpiphp: Slot [4] registered [ 0.363117] acpiphp: Slot [11] registered [ 0.364195] acpiphp: Slot [12] registered [ 0.365113] acpiphp: Slot [13] registered [ 0.366107] acpiphp: Slot [14] registered [ 0.367000] acpiphp: Slot [15] registered [ 0.368106] acpiphp: Slot [16] registered [ 0.370066] acpiphp: Slot [17] registered [ 0.371092] acpiphp: Slot [18] registered [ 0.373090] acpiphp: Slot [19] registered [ 0.374065] acpiphp: Slot [20] registered [ 0.375376] acpiphp: Slot [21] registered [ 0.378301] acpiphp: Slot [22] registered [ 0.380123] acpiphp: Slot [23] registered [ 0.381084] acpiphp: Slot [24] registered [ 0.382000] acpiphp: Slot [25] registered [ 0.382000] acpiphp: Slot [26] registered [ 0.383231] acpiphp: Slot [27] registered [ 0.385098] acpiphp: Slot [28] registered [ 0.386167] acpiphp: Slot [29] registered [ 0.387232] acpiphp: Slot [30] registered [ 0.388228] acpiphp: Slot [31] registered [ 0.389151] PCI host bridge to bus 0000:00 [ 0.390173] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.392063] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.395077] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.398131] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.399000] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.399056] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.401662] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.402000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.404318] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.413018] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.417846] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.420042] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.422022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.425028] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.427708] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.430755] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.433041] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.435074] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.440016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.453024] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.460028] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.464809] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.470018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.477023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.493020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.500000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.502019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.507024] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.518032] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.525463] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.537061] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.556029] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.586221] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.604000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.615021] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.621016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.635035] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.645456] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.650013] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.657024] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.667019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.678464] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.685017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.694019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.714023] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.726093] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.727000] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.727409] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.729468] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.730337] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.738725] iommu: Default domain type: Passthrough [ 0.739478] SCSI subsystem initialized [ 0.741156] ACPI: bus type USB registered [ 0.743119] usbcore: registered new interface driver usbfs [ 0.745082] usbcore: registered new interface driver hub [ 0.747082] usbcore: registered new device driver usb [ 0.748231] pps_core: LinuxPPS API ver. 1 registered [ 0.749000] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.751069] PTP clock support registered [ 0.754321] EDAC MC: Ver: 3.0.0 [ 0.755490] PCI: Using ACPI for IRQ routing [ 0.756000] NetLabel: Initializing [ 0.758010] NetLabel: domain hash size = 128 [ 0.760007] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.762092] NetLabel: unlabeled traffic allowed by default [ 0.765019] vgaarb: loaded [ 0.766419] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.768013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.779589] clocksource: Switched to clocksource kvm-clock [ 0.904412] VFS: Disk quotas dquot_6.6.0 [ 0.906384] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.908971] *** VALIDATE ramfs *** [ 0.909948] *** VALIDATE hugetlbfs *** [ 0.912156] pnp: PnP ACPI init [ 0.915689] pnp: PnP ACPI: found 6 devices [ 0.935121] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.939149] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.941101] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.943223] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.945484] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.947530] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.951079] NET: Registered protocol family 2 [ 0.953399] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.958497] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.961809] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.966820] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.970494] TCP: Hash tables configured (established 65536 bind 65536) [ 0.973691] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.977230] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.980330] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.983422] NET: Registered protocol family 1 [ 0.986682] RPC: Registered named UNIX socket transport module. [ 0.988658] RPC: Registered udp transport module. [ 0.990135] RPC: Registered tcp transport module. [ 0.992112] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.994123] NET: Registered protocol family 44 [ 0.995668] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.997638] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.000137] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.002329] PCI: CLS 0 bytes, default 64 [ 1.004160] Unpacking initramfs... [ 2.765174] debug: unmapping init [mem 0xffff91af7cc54000-0xffff91af7ffbffff] [ 2.769305] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.771297] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.773981] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.848260] Initialise system trusted keyrings [ 3.852252] Key type blacklist registered [ 3.856636] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.880965] zbud: loaded [ 3.888850] *** VALIDATE nfs *** [ 3.891364] *** VALIDATE nfs4 *** [ 3.894604] pstore: using deflate compression [ 3.903950] Platform Keyring initialized [ 4.298918] NET: Registered protocol family 38 [ 4.302701] Key type asymmetric registered [ 4.305480] Asymmetric key parser 'x509' registered [ 4.312862] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.316566] io scheduler mq-deadline registered [ 4.317746] io scheduler kyber registered [ 4.323873] io scheduler bfq registered [ 4.329158] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.336348] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.343965] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.350473] ACPI: Power Button [PWRF] [ 4.361662] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.375668] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.403342] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.412355] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.433191] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.467933] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.509863] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.518297] Non-volatile memory driver v1.3 [ 4.520700] Linux agpgart interface v0.103 [ 4.570627] virtio_blk virtio1: [vda] 134008 512-byte logical blocks (68.6 MB/65.4 MiB) [ 4.574458] vda: detected capacity change from 0 to 68612096 [ 4.595137] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.597966] vdb: detected capacity change from 0 to 1073741824 [ 4.619515] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.622905] vdc: detected capacity change from 0 to 2621440000 [ 4.642678] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.647204] vdd: detected capacity change from 0 to 2621440000 [ 4.785805] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.788849] vde: detected capacity change from 0 to 4294967296 [ 4.815572] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.820591] vdf: detected capacity change from 0 to 4294967296 [ 4.832502] libphy: Fixed MDIO Bus: probed [ 4.843307] usbcore: registered new interface driver usbserial_generic [ 4.849078] usbserial: USB Serial support registered for generic [ 4.854413] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.863557] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.867445] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.873973] mousedev: PS/2 mouse device common for all mice [ 4.879648] rtc_cmos 00:05: RTC can wake from S4 [ 4.884610] rtc_cmos 00:05: registered as rtc0 [ 4.886345] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.895283] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.895344] intel_pstate: CPU model not supported [ 4.906939] hid: raw HID events driver (C) Jiri Kosina [ 5.117417] usbcore: registered new interface driver usbhid [ 5.119263] usbhid: USB HID core driver [ 5.120850] drop_monitor: Initializing network drop monitor service [ 5.124459] Initializing XFRM netlink socket [ 5.146133] NET: Registered protocol family 10 [ 5.163816] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.175337] Segment Routing with IPv6 [ 5.186115] NET: Registered protocol family 17 [ 5.187557] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.192147] mpls_gso: MPLS GSO support [ 5.277276] RAS: Correctable Errors collector initialized. [ 5.279376] AVX version of gcm_enc/dec engaged. [ 5.281875] AES CTR mode by8 optimization enabled [ 5.480745] sched_clock: Marking stable (5480671358, 0)->(6577740650, -1097069292) [ 5.484237] registered taskstats version 1 [ 5.486445] Loading compiled-in X.509 certificates [ 5.488582] zswap: loaded using pool lzo/zbud [ 5.518919] Key type big_key registered [ 5.533336] Key type encrypted registered [ 5.534892] ima: No TPM chip found, activating TPM-bypass! [ 5.537236] ima: Allocated hash algorithm: sha1 [ 5.538979] ima: No architecture policies found [ 5.540174] evm: Initialising EVM extended attributes: [ 5.542180] evm: security.selinux [ 5.543438] evm: security.ima [ 5.547295] evm: security.capability [ 5.551516] evm: HMAC attrs: 0x1 [ 5.558120] rtc_cmos 00:05: setting system clock to 2026-04-22 14:52:21 UTC (1776869541) [ 5.577991] debug: unmapping init [mem 0xffffffff9f203000-0xffffffff9f3fffff] [ 5.585405] debug: unmapping init [mem 0xffffffff9df82000-0xffffffff9e258fff] [ 5.595140] Write protecting the kernel read-only data: 28672k [ 5.604233] debug: unmapping init [mem 0xffffffff9c603000-0xffffffff9c7fffff] [ 5.609925] debug: unmapping init [mem 0xffffffff9cf14000-0xffffffff9cffffff] [ 5.715394] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.730681] systemd[1]: Detected virtualization kvm. [ 5.732798] systemd[1]: Detected architecture x86-64. [ 5.734559] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.756236] systemd[1]: No hostname configured. [ 5.757837] systemd[1]: Set hostname to . [ 5.759877] random: systemd: uninitialized urandom read (16 bytes read) [ 5.762255] systemd[1]: Initializing machine ID from random generator. [ 6.026072] random: ln: uninitialized urandom read (6 bytes read) [ 6.327108] random: systemd: uninitialized urandom read (16 bytes read) [ 6.356087] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 6.364395] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 6.369352] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. [ OK ] Reached target Slices. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Swap. Starting Journal Service... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.551458] device-mapper: uevent: version 1.0.3 [ 7.553772] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 9.108227] virtio_net virtio0 ens2: renamed from eth0 [ 10.259198] random: fast init done [ 15.378421] scsi host0: ata_piix [ 15.413198] scsi host1: ata_piix [ 15.416091] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 15.418977] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 17.235898] random: crng init done [ 17.237352] random: 7 urandom warning(s) missed due to ratelimiting [ 19.562774] dracut-initqueue[583]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 21.409961] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 23.271269] printk: systemd: 26 output lines suppressed due to ratelimiting [ 23.842909] SELinux: Disabled at runtime. [ 23.915425] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 23.924701] systemd[1]: Detected virtualization kvm. [ 23.926649] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 26.000651] systemd[1]: initrd-switch-root.service: Succeeded. [ 26.006388] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 26.014475] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 26.019354] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 26.023474] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 26.035024] systemd[1]: Starting Journal Service... Starting Journal Service... [ 26.083458] systemd[1]: Starting Apply Kernel Variables... Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. Mounting Kernel Debug File System... [ OK ] Reached target rpc_pipefs.target. Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Control Socket. Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on RPCbind Server Activation Socket. [ 26.341410] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target RPC Port Mapper. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting POSIX Message Queue File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 28.227206] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 30.395738] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 30.407046] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 31.367501] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 32.638813] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit)[ 35.745317] Key type dns_resolver registered [ *** ] A start job is running for Configur…only root support (10s / no limit)[ 36.163462] NFS: Registering the id_resolver key type [ 36.166727] Key type id_resolver registered [ 36.168279] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg246-server login: [ 62.725188] libcfs: loading out-of-tree module taints kernel. [ 62.738896] Key type ._llcrypt registered [ 62.739892] Key type .llcrypt registered [ 62.776944] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 70.017176] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 70.561456] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 70.566651] alg: No test for adler32 (adler32-zlib) [ 71.522386] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 71.788171] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 73.391142] Key type lgssc registered [ 74.007269] Lustre: Echo OBD driver; http://www.lustre.org/ [ 80.955247] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 96.544392] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 101.483910] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 101.493598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 102.607899] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 102.624522] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 102.667170] Lustre: lustre-MDT0000: new disk, initializing [ 102.714840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 102.722392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 104.218907] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 109.895756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 109.930539] Lustre: 6474:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 109.941643] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 109.943750] Lustre: Skipped 1 previous similar message [ 109.984086] Lustre: lustre-MDT0001: new disk, initializing [ 110.006047] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 110.015367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 110.018854] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 111.382485] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 113.793286] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 117.328512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 117.421690] Lustre: lustre-OST0000: new disk, initializing [ 117.423796] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 117.445778] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 118.957066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 118.960122] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 118.995744] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 119.441995] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 124.998336] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 125.055966] Lustre: lustre-OST0001: new disk, initializing [ 125.059535] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 125.082873] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 127.351882] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 127.665447] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 127.669466] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 127.691747] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 133.890425] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 141.901252] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 149.082752] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing check_logdir /tmp/testlogs/ [ 150.903070] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing yml_node [ 152.660109] Lustre: DEBUG MARKER: Client: 2.17.52.53 [ 153.631428] Lustre: DEBUG MARKER: MDS: 2.17.52.53 [ 154.581789] Lustre: DEBUG MARKER: OSS: 2.17.52.53 [ 155.266808] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Apr 22 10:54:50 EDT 2026 [ 161.681087] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 162.275346] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 174.048650] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 174.052481] Lustre: Skipped 1 previous similar message [ 174.054700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 179.167769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 179.170283] Lustre: Skipped 6 previous similar messages [ 184.287593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 184.289871] Lustre: Skipped 3 previous similar messages [ 188.383098] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 188.450407] Lustre: server umount lustre-MDT0000 complete [ 189.724739] LustreError: 9431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776869725 with bad export cookie 8624428595337120944 [ 189.726046] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 189.730920] LustreError: 9431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 194.527521] LustreError: 6482:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 194.527582] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 194.534024] LustreError: 6482:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 6 previous similar messages [ 194.539659] Lustre: Skipped 2 previous similar messages [ 194.547489] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 194.551254] Lustre: Skipped 1 previous similar message [ 199.647519] LustreError: 6481:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 199.647571] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 199.654145] LustreError: 6481:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 199.658872] Lustre: Skipped 1 previous similar message [ 204.255092] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 204.331284] Lustre: server umount lustre-MDT0001 complete [ 205.866712] Lustre: server umount lustre-OST0000 complete [ 207.344591] Lustre: server umount lustre-OST0001 complete [ 209.596699] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 212.715466] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 225.390322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 225.507381] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 225.520746] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 225.572187] Lustre: lustre-MDT0000: new disk, initializing [ 225.622549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 225.630933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 227.086315] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 231.842741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 231.889583] Lustre: 14936:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 232.027684] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 233.537778] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 237.126434] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 238.862816] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 241.866569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 241.988797] Lustre: lustre-OST0000: new disk, initializing [ 241.991051] Lustre: Skipped 1 previous similar message [ 241.994739] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 241.997443] Lustre: Skipped 2 previous similar messages [ 242.033753] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 242.037859] Lustre: Skipped 1 previous similar message [ 244.033441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 244.036299] Lustre: Skipped 1 previous similar message [ 244.038338] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 244.083089] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 244.251181] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 247.742607] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 249.328686] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 249.414413] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 250.800056] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 250.872528] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 254.436704] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 254.440038] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 254.444772] Lustre: Skipped 1 previous similar message [ 254.446989] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 254.451591] Lustre: Skipped 1 previous similar message [ 257.620758] Lustre: server umount lustre-OST0000 complete [ 259.552481] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 259.558416] Lustre: Skipped 2 previous similar messages [ 265.305232] Lustre: server umount lustre-MDT0000 complete [ 266.888518] LustreError: 16679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776869802 with bad export cookie 8624428595337122407 [ 266.892926] LustreError: 16679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 266.893509] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 267.025165] Lustre: server umount lustre-MDT0001 complete [ 269.445679] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 10:56:44 (1776869804) [ 277.665441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 277.890462] LustreError: 19402:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 277.920918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 279.736569] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 283.104769] LustreError: 19403:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 283.642862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 285.445952] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 287.708713] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 289.262456] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 290.220203] Lustre: Failing over lustre-MDT0000 [ 290.339974] Lustre: server umount lustre-MDT0000 complete [ 293.280423] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 293.281437] LustreError: 19397:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 293.285169] Lustre: Skipped 1 previous similar message [ 303.585487] LustreError: 19397:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 303.595274] LustreError: 19397:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 304.673763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 304.740964] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 306.527871] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 308.964328] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 308.968839] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 308.984482] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 310.620644] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 314.336695] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 314.338393] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 314.341382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 314.341388] Lustre: Skipped 4 previous similar messages [ 314.344393] Lustre: Skipped 1 previous similar message [ 314.357965] LustreError: Skipped 1 previous similar message [ 319.079091] Lustre: server umount lustre-MDT0000 complete [ 320.659465] LustreError: 19383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776869856 with bad export cookie 8624428595337123821 [ 320.662705] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 320.663610] LustreError: 19383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 320.790525] Lustre: server umount lustre-MDT0001 complete [ 324.495306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 324.666776] LustreError: 22393:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 324.673291] LustreError: 22393:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 324.703321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 324.706853] Lustre: Skipped 2 previous similar messages [ 326.200067] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 330.243272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 332.209303] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 334.564584] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 336.277056] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 337.292945] Lustre: Failing over lustre-MDT0000 [ 337.388743] Lustre: server umount lustre-MDT0000 complete [ 339.936338] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 339.941973] Lustre: Skipped 1 previous similar message [ 351.060405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 351.112843] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 352.794799] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 355.296357] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 355.299908] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 355.302375] Lustre: Skipped 1 previous similar message [ 355.307428] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 356.339773] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 358.481238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 358.483952] Lustre: Skipped 2 previous similar messages [ 364.637471] Lustre: server umount lustre-MDT0000 complete [ 365.995378] LustreError: 22375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776869901 with bad export cookie 8624428595337124759 [ 365.997822] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 366.000116] LustreError: 22375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 366.134150] Lustre: server umount lustre-MDT0001 complete [ 369.920336] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 10:58:24 (1776869904) [ 376.290795] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 380.084471] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 396.686601] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 401.235341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 401.341649] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 401.356548] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 401.395254] Lustre: lustre-MDT0000: new disk, initializing [ 401.427116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 401.429662] Lustre: Skipped 2 previous similar messages [ 401.437111] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 402.980546] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 406.568007] hrtimer: interrupt took 2999394 ns [ 408.010776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 408.059679] Lustre: 28366:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 408.178975] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 409.821278] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 413.393983] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 414.819654] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 417.492151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 419.161283] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 419.179663] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 419.186462] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 419.897889] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 423.380386] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 426.175810] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 434.657318] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 434.657891] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 434.660588] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 434.665031] Lustre: Skipped 3 previous similar messages [ 434.667525] Lustre: Skipped 2 previous similar messages [ 437.571855] Lustre: server umount lustre-OST0000 complete [ 445.410187] Lustre: server umount lustre-MDT0000 complete [ 446.890146] LustreError: 28359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776869982 with bad export cookie 8624428595337125165 [ 446.893782] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 446.895363] LustreError: 28359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 447.077722] Lustre: server umount lustre-MDT0001 complete [ 450.939167] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 452.065716] Key type lgssc unregistered [ 452.206542] LNet: 31928:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 452.209923] LNetError: 31928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 452.223343] LNet: Removed LNI 192.168.202.146@tcp [ 452.599174] Key type .llcrypt unregistered [ 452.601549] Key type ._llcrypt unregistered [ 462.919619] Key type ._llcrypt registered [ 462.920929] Key type .llcrypt registered [ 462.985025] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 463.392860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 463.420061] alg: No test for adler32 (adler32-zlib) [ 464.315695] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 464.419042] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 466.015205] Key type lgssc registered [ 466.477311] Lustre: Echo OBD driver; http://www.lustre.org/ [ 469.897596] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 469.914587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 471.131879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 472.643777] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 476.120938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 476.272504] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 477.716133] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 479.708030] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 481.019338] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 483.486186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 483.616968] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 485.886287] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 486.694806] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 488.043992] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 505.416814] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 11:00:40 (1776870040) [ 506.244365] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 507.020589] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 509.919722] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 11:00:44 (1776870044) [ 513.469826] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 11:00:48 (1776870048) [ 513.979649] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 517.892715] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 517.895603] Lustre: Skipped 2 previous similar messages [ 518.705454] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 11:00:53 (1776870053) [ 522.466168] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 522.469679] Lustre: Skipped 4 previous similar messages [ 528.825887] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 11:01:03 (1776870063) [ 529.952951] Lustre: *** cfs_fail_loc=131b, val=2*** [ 531.300601] Lustre: *** cfs_fail_loc=131b, val=2*** [ 534.427921] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 11:01:09 (1776870069) [ 535.244152] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 535.246182] Lustre: Skipped 9 previous similar messages [ 540.941334] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 11:01:15 (1776870075) [ 548.320861] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 548.323496] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 548.331536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 548.334243] Lustre: Skipped 1 previous similar message [ 552.081084] Lustre: server umount lustre-OST0000 complete [ 558.048111] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 558.051663] LustreError: Skipped 1 previous similar message [ 558.054153] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 558.060457] Lustre: Skipped 1 previous similar message [ 558.062591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 563.681823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 563.685188] Lustre: Skipped 2 previous similar messages [ 568.287113] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 568.360480] Lustre: server umount lustre-MDT0000 complete [ 568.800344] LustreError: 33297:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 568.808282] LustreError: 33297:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 569.923893] LustreError: 33283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776870105 with bad export cookie 11883628585374696165 [ 569.926577] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 569.930543] LustreError: 33283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 570.064609] Lustre: server umount lustre-MDT0001 complete [ 574.523874] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 575.746696] Key type lgssc unregistered [ 575.890198] LNet: 40814:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 575.892977] LNetError: 40814:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 575.899693] LNet: Removed LNI 192.168.202.146@tcp [ 576.272355] Key type .llcrypt unregistered [ 576.273613] Key type ._llcrypt unregistered [ 583.613309] Key type ._llcrypt registered [ 583.614335] Key type .llcrypt registered [ 583.664168] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 590.568829] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 590.895341] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 590.912051] alg: No test for adler32 (adler32-zlib) [ 591.793681] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 591.890586] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 593.487709] Key type lgssc registered [ 593.948889] Lustre: Echo OBD driver; http://www.lustre.org/ [ 609.029074] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 609.036337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 610.132097] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 610.147138] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 610.185798] Lustre: lustre-MDT0000: new disk, initializing [ 610.216699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 610.224959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 611.726293] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 616.485873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 616.530862] Lustre: 43907:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 616.554415] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 616.556069] Lustre: Skipped 1 previous similar message [ 616.593499] Lustre: lustre-MDT0001: new disk, initializing [ 616.632456] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 616.647723] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 616.650765] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 618.100738] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 621.738179] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 623.197347] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 625.966474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 626.100699] Lustre: lustre-OST0000: new disk, initializing [ 626.102893] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 626.141944] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 628.326718] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 631.999548] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 632.833099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 632.839456] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 632.874759] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 633.681461] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 633.759069] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 635.329499] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 635.412603] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 637.921167] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 637.921800] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 637.925058] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 637.931110] Lustre: Skipped 1 previous similar message [ 642.263633] Lustre: server umount lustre-OST0000 complete [ 647.648326] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 647.652272] LustreError: Skipped 1 previous similar message [ 647.653622] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 647.659248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 647.662085] Lustre: Skipped 1 previous similar message [ 653.281124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 653.285051] Lustre: Skipped 3 previous similar messages [ 657.376716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 657.379839] Lustre: Skipped 1 previous similar message [ 658.400653] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 658.478375] Lustre: server umount lustre-MDT0000 complete [ 660.069298] LustreError: 43900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776870196 with bad export cookie 3127542180399067868 [ 660.075075] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 660.078123] LustreError: 43900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 660.218856] Lustre: server umount lustre-MDT0001 complete [ 664.048171] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 11:03:19 (1776870199) [ 666.534432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 666.726126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 668.158064] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 671.484848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 671.618203] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 673.044741] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 675.009138] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 676.324167] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 678.823577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 681.130823] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 683.357208] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 696.064567] LustreError: 50167:0:(llog.c:444:llog_init_handle()) llog type is not specified! [ 696.418821] Lustre: Modifying parameter general.timeout=1 in log params [ 697.734232] Lustre: Modifying parameter general.timeout=5 in log params [ 697.736618] Lustre: Skipped 3 previous similar messages [ 700.000493] Lustre: Modifying parameter general.timeout=12 in log params [ 700.003306] Lustre: Skipped 6 previous similar messages [ 704.036524] Lustre: Modifying parameter general.timeout=24 in log params [ 704.038444] Lustre: Skipped 11 previous similar messages [ 712.073810] Lustre: Modifying parameter general.timeout=49 in log params [ 712.075947] Lustre: Skipped 24 previous similar messages [ 728.329881] Lustre: Modifying parameter general.timeout=97 in log params [ 728.331836] Lustre: Skipped 47 previous similar messages [ 743.658543] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 11:04:38 (1776870278) [ 744.668832] Lustre: 57072:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 746.076885] Lustre: 57268:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 746.085833] Lustre: 57268:0:(mgs_llog.c:1437:mgs_modify_param()) Skipped 1 previous similar message [ 748.323474] Lustre: 57562:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 748.327311] Lustre: 57562:0:(mgs_llog.c:1437:mgs_modify_param()) Skipped 2 previous similar messages [ 757.744548] Lustre: 58911:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 757.749548] LustreError: 58911:0:(mgs_handler.c:1202:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 759.165656] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 11:04:54 (1776870294) [ 760.424646] Lustre: Setting parameter general.jobid_var=TESTNAME in log params [ 760.426806] Lustre: Skipped 54 previous similar messages [ 763.872323] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 763.874975] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 763.879112] Lustre: Skipped 1 previous similar message [ 763.880857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 769.050408] Lustre: server umount lustre-MDT0000 complete [ 770.016716] LustreError: 47959:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 770.022155] LustreError: 47959:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 770.323883] LustreError: 49574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776870306 with bad export cookie 3127542180399068750 [ 770.326902] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 770.327920] LustreError: 49574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 770.429164] Lustre: server umount lustre-MDT0001 complete [ 781.918092] Lustre: server umount lustre-OST0000 complete [ 791.571649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 791.673824] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 791.794158] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 791.796167] Lustre: Skipped 1 previous similar message [ 793.148966] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 796.027053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 796.061837] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 796.070651] Lustre: 61433:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 797.584765] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 799.493398] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 800.795292] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 803.216045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 803.288676] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 805.251104] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 807.187935] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 813.241439] Lustre: 63659:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 818.145582] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 818.146450] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 818.147861] LustreError: Skipped 1 previous similar message [ 818.151448] Lustre: Skipped 2 previous similar messages [ 818.152139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 818.158217] Lustre: Skipped 4 previous similar messages [ 821.202987] Lustre: server umount lustre-OST0000 complete [ 823.264882] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 823.271872] Lustre: Skipped 2 previous similar messages [ 828.636805] Lustre: server umount lustre-MDT0000 complete [ 829.970333] LustreError: 61424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776870365 with bad export cookie 3127542180399071060 [ 829.973226] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 829.975502] LustreError: 61424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 830.098570] Lustre: server umount lustre-MDT0001 complete [ 833.805593] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 834.853359] Key type lgssc unregistered [ 834.988370] LNet: 64848:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 834.990961] LNetError: 64848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 834.997466] LNet: Removed LNI 192.168.202.146@tcp [ 835.307609] Key type .llcrypt unregistered [ 835.309281] Key type ._llcrypt unregistered [ 844.285159] Key type ._llcrypt registered [ 844.286114] Key type .llcrypt registered [ 844.324678] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 844.742845] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 844.750520] alg: No test for adler32 (adler32-zlib) [ 845.595534] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 845.677340] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 847.255141] Key type lgssc registered [ 847.610056] Lustre: Echo OBD driver; http://www.lustre.org/ [ 850.158247] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 850.169443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 851.358549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 852.724658] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 855.592029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 855.788325] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 857.080617] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 858.895735] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 860.060814] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 862.320339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 862.495366] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 864.276212] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 866.076231] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 884.729665] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 888.288938] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 888.290029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 888.292459] Lustre: Skipped 2 previous similar messages [ 888.294052] Lustre: Skipped 1 previous similar message [ 893.197871] Lustre: server umount lustre-MDT0000 complete [ 893.408300] LustreError: 66226:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 893.413076] LustreError: 66226:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 4 previous similar messages [ 894.375024] LustreError: 66207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776870430 with bad export cookie 15417845066321847986 [ 894.377713] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 894.378199] LustreError: 66207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 894.479439] Lustre: server umount lustre-MDT0001 complete [ 914.949775] Lustre: server umount lustre-OST0000 complete [ 923.929385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 924.013267] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 924.022949] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 924.106172] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 925.363473] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 928.188615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 928.224472] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 928.233790] Lustre: 70780:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 928.379920] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 929.670929] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 931.501661] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 932.688618] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 935.004429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 935.067886] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 937.154396] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 938.948727] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 944.810942] Lustre: 73016:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 945.421784] Lustre: Modifying parameter general.jobid_var=disable in log params [ 947.170312] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 947.171194] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 947.173530] LustreError: Skipped 1 previous similar message [ 947.178578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 947.181885] Lustre: Skipped 2 previous similar messages [ 948.758181] Lustre: server umount lustre-OST0000 complete [ 952.082770] Lustre: server umount lustre-MDT0000 complete [ 952.288297] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 952.288769] LustreError: 70786:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 952.294577] Lustre: Skipped 2 previous similar messages [ 952.300859] LustreError: 70786:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 953.314258] LustreError: 70771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776870489 with bad export cookie 15417845066321848777 [ 953.317233] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 953.317364] LustreError: 70771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 953.423929] Lustre: server umount lustre-MDT0001 complete [ 956.967980] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 957.921922] Key type lgssc unregistered [ 958.046398] LNet: 74202:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 958.049750] LNetError: 74202:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 958.058311] LNet: Removed LNI 192.168.202.146@tcp [ 958.336063] Key type .llcrypt unregistered [ 958.337131] Key type ._llcrypt unregistered [ 967.076140] Key type ._llcrypt registered [ 967.076964] Key type .llcrypt registered [ 967.115358] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 967.455202] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 967.492573] alg: No test for adler32 (adler32-zlib) [ 968.338494] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 968.421410] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 969.999140] Key type lgssc registered [ 970.338302] Lustre: Echo OBD driver; http://www.lustre.org/ [ 972.874258] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 972.880086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 974.042029] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 975.332583] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 978.195471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 978.405843] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 979.730329] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 981.658434] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 982.857715] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 985.080393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 985.257487] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 987.173986] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 989.084587] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1001.830202] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1002.421823] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1002.423782] Lustre: Skipped 1 previous similar message [ 1003.595547] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1003.597760] Lustre: Skipped 3 previous similar messages [ 1005.665286] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1005.667564] Lustre: Skipped 6 previous similar messages [ 1009.787675] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1009.790441] Lustre: Skipped 13 previous similar messages [ 1018.061184] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1018.063070] Lustre: Skipped 27 previous similar messages [ 1033.683324] Lustre: 83320:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1034.535193] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 11:09:29 (1776870569) [ 1035.065686] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1035.687757] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 11:09:30 (1776870570) [ 1039.841697] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1039.843504] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1039.844260] LustreError: Skipped 1 previous similar message [ 1039.844942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1039.849286] Lustre: Skipped 1 previous similar message [ 1042.699359] Lustre: server umount lustre-OST0000 complete [ 1044.960678] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1044.961118] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1044.963781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1044.967160] Lustre: Skipped 1 previous similar message [ 1044.970552] Lustre: Skipped 2 previous similar messages [ 1050.080109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1050.082425] Lustre: Skipped 1 previous similar message [ 1055.200070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1055.202370] Lustre: Skipped 1 previous similar message [ 1058.271110] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1058.332447] Lustre: server umount lustre-MDT0000 complete [ 1059.296986] LustreError: 75581:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1059.303844] LustreError: 75581:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 1059.559196] LustreError: 75561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776870595 with bad export cookie 5438484735525032407 [ 1059.561698] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1059.562808] LustreError: 75561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1059.673981] Lustre: server umount lustre-MDT0001 complete [ 1063.067898] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 1064.027271] Key type lgssc unregistered [ 1064.154348] LNet: 84550:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1064.156915] LNetError: 84550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1064.164276] LNet: Removed LNI 192.168.202.146@tcp [ 1064.460188] Key type .llcrypt unregistered [ 1064.461188] Key type ._llcrypt unregistered [ 1065.774663] Key type ._llcrypt registered [ 1065.775480] Key type .llcrypt registered [ 1065.810254] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1067.033786] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules [ 1067.316634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1067.358751] alg: No test for adler32 (adler32-zlib) [ 1068.219259] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 1068.296796] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 1069.871157] Key type lgssc registered [ 1070.215219] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1070.567566] LustreError: 85670:0:(lu_object.c:1477:lu_context_key_register()) cfs_race id 60d sleeping [ 1078.032134] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 1081.122136] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1091.866523] LustreError: 85670:0:(lu_object.c:1477:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1091.874049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1093.049700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1094.255055] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1096.931674] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 11:10:32 (1776870632) [ 1098.405887] LustreError: 86707:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1098.410353] LustreError: 86707:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1098.414359] LustreError: 86707:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 0, failed: rc = -5 [ 1098.532683] Lustre: server umount lustre-MDT0000 complete [ 1102.755361] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 1103.738038] Key type lgssc unregistered [ 1103.852337] LNet: 88163:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1103.854398] LNetError: 88163:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1103.864338] LNet: Removed LNI 192.168.202.146@tcp [ 1104.155672] Key type .llcrypt unregistered [ 1104.156748] Key type ._llcrypt unregistered [ 1111.125955] Key type ._llcrypt registered [ 1111.126755] Key type .llcrypt registered [ 1111.163478] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 1111.659155] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1111.666801] alg: No test for adler32 (adler32-zlib) [ 1112.515295] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 1112.590341] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 1114.167133] Key type lgssc registered [ 1114.516172] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1117.020687] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1117.025484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1118.177013] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1119.391582] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1122.061160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1122.261171] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1123.455356] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1125.066962] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1126.164436] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1128.366940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1128.417627] LustreError: 90940:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1128.425382] LustreError: 90940:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 1128.589763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1130.406104] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1132.084079] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1158.886570] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 11:11:34 (1776870694) [ 1159.412770] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1159.944532] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 11:11:35 (1776870695) [ 1163.231820] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1163.234867] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1163.239208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1163.745671] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1163.750482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1163.753249] Lustre: Skipped 1 previous similar message [ 1167.057956] Lustre: server umount lustre-MDT0000 complete [ 1168.228989] LustreError: 90966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776870704 with bad export cookie 10651567307449673339 [ 1168.232789] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1168.233256] LustreError: 90966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1168.332601] Lustre: server umount lustre-MDT0001 complete [ 1179.834430] Lustre: server umount lustre-OST0000 complete [ 1183.611569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1183.758487] LustreError: 93273:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1183.811128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1185.025421] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1187.804412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1188.013691] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1189.238270] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1190.944347] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1192.093257] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1196.054827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1196.126699] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1196.129290] LustreError: 93262:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1196.137701] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1196.140185] LustreError: 95130:0:(tgt_mount.c:2564:server_fill_super()) Unable to start targets: -98 [ 1196.142223] LustreError: 95130:0:(tgt_mount.c:2082:server_put_super()) no obd lustre-OST0000 [ 1196.143971] LustreError: 95130:0:(tgt_mount.c:114:server_deregister_mount()) lustre-OST0000 not registered [ 1196.164192] Lustre: server umount lustre-OST0000 complete [ 1196.165601] LustreError: 95130:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1198.863666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1198.935617] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1198.938878] LustreError: 93262:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1198.943257] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1198.945748] LustreError: 95591:0:(tgt_mount.c:2564:server_fill_super()) Unable to start targets: -98 [ 1198.947803] LustreError: 95591:0:(tgt_mount.c:2082:server_put_super()) no obd lustre-OST0000 [ 1198.949373] LustreError: 95591:0:(tgt_mount.c:114:server_deregister_mount()) lustre-OST0000 not registered [ 1198.973224] Lustre: server umount lustre-OST0000 complete [ 1198.975112] LustreError: 95591:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1202.079419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1202.137896] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1202.140235] Lustre: Found index 0 for lustre-OST0000, updating log [ 1202.142657] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1202.171040] Lustre: lustre-OST0000: new disk, initializing [ 1202.173435] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1202.266268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1204.111513] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1205.392488] Lustre: Failing over lustre-OST0000 [ 1205.437495] Lustre: server umount lustre-OST0000 complete [ 1206.653705] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1206.654225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1206.657086] Lustre: Skipped 2 previous similar messages [ 1211.871925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1211.874218] Lustre: Skipped 2 previous similar messages [ 1212.815551] Lustre: server umount lustre-MDT0000 complete [ 1212.896276] LustreError: 93274:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1212.901043] LustreError: 93274:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 1213.969129] LustreError: 96124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776870749 with bad export cookie 10651567307449676069 [ 1213.971839] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1213.971927] LustreError: 96124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1214.068812] Lustre: server umount lustre-MDT0001 complete [ 1216.761175] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 11:12:31 (1776870751) [ 1221.899302] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 1225.277699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1225.425781] LustreError: 98427:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1225.457214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1226.749015] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1229.561360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1231.078373] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1232.063059] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 1234.229021] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1236.221868] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1238.954574] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1238.983182] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 1239.015688] Lustre: lustre-OST0001: new disk, initializing [ 1239.017317] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1239.419324] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1239.422155] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1239.460471] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1240.900382] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1243.892099] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1251.513091] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1260.000307] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 1260.002680] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1260.006642] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1271.266179] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1271.268608] Lustre: Skipped 10 previous similar messages [ 1272.287130] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1272.342043] Lustre: server umount lustre-MDT0001 complete [ 1272.650613] LustreError: 99139:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1275.250932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1275.281633] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1275.283338] Lustre: Found index 1 for lustre-MDT0001, updating log [ 1275.296903] Lustre: 98417:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1275.501903] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1275.504204] Lustre: Skipped 3 previous similar messages [ 1276.706797] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1277.093133] LustreError: 102488:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 1277.095314] LustreError: 102488:0:(tgt_mount.c:666:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 1277.097649] LustreError: 98419:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 1277.121463] LustreError: 102493:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.146@tcp: cfg command failed: rc = -17 [ 1277.124581] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1277.124581] [ 1279.703611] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID ========================================================== 11:13:34 (1776870814) [ 1280.998229] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1285.276969] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 1290.854376] Lustre: 104250:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1298.389060] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1299.645685] Lustre: 105381:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1352.159637] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1352.162297] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1352.166091] Lustre: Skipped 2 previous similar messages [ 1352.167657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1356.512528] Lustre: server umount lustre-MDT0000 complete [ 1357.696984] LustreError: 98407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776870893 with bad export cookie 10651567307449676937 [ 1357.698606] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1357.700500] LustreError: 98407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1357.791600] LustreError: 98422:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1357.797531] LustreError: 98422:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 1357.820510] Lustre: server umount lustre-MDT0001 complete [ 1365.184902] Lustre: server umount lustre-OST0000 complete [ 1366.498544] Lustre: server umount lustre-OST0001 complete [ 1368.357634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1371.425094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1374.312856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1377.497458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1383.531693] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 1387.005257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1387.014554] Lustre: lustre-MDT0000: reset Object Index mappings [ 1387.189558] LustreError: 109283:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1387.218073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1388.549963] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1391.407973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1391.417848] Lustre: lustre-MDT0001: reset Object Index mappings [ 1392.941377] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1393.902383] Lustre: 110398:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1395.969456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1397.221913] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:354 to 0x280000401:386) [ 1397.926711] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1400.563534] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1401.768610] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 1401.769205] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:195 to 0x280000400:225) [ 1401.770043] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 1402.470652] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1405.498695] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1406.688519] Lustre: 112237:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1425.259090] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 11:16:00 (1776870960) [ 1458.146480] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1458.148383] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1458.148802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1458.148807] Lustre: Skipped 3 previous similar messages [ 1458.150220] Lustre: Skipped 8 previous similar messages [ 1460.824707] Lustre: server umount lustre-MDT0000 complete [ 1461.904895] LustreError: 109264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776870997 with bad export cookie 10651567307449723704 [ 1461.908394] LustreError: 109264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1461.909613] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1463.263986] LustreError: 109279:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1463.268321] LustreError: 109279:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 1468.154759] Lustre: server umount lustre-MDT0001 complete [ 1475.522840] Lustre: server umount lustre-OST0000 complete [ 1482.839867] Lustre: server umount lustre-OST0001 complete [ 1484.707486] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 1487.879597] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 1500.068900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1500.154811] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1500.164547] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1500.200483] Lustre: lustre-MDT0000: new disk, initializing [ 1500.226948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1501.476105] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1504.453593] Lustre: server umount lustre-MDT0000 complete [ 1507.589102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1507.686861] Lustre: Found index 0 for lustre-MDT0000, updating log [ 1507.691324] Lustre: 116442:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1507.697771] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 1509.138022] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1510.865699] Lustre: server umount lustre-MDT0000 complete [ 1513.822848] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 11:17:28 (1776871048) [ 1514.392247] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 1514.973879] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 11:17:30 (1776871050) [ 1520.667157] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 1523.935258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1524.084510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1524.087055] Lustre: Skipped 5 previous similar messages [ 1525.320602] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1527.983755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1528.021070] Lustre: 118458:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1528.043169] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1528.045099] Lustre: Skipped 1 previous similar message [ 1528.082481] Lustre: lustre-MDT0001: new disk, initializing [ 1528.126590] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1528.129464] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1529.349669] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1531.543072] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 1533.676674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1533.779683] Lustre: lustre-OST0000: new disk, initializing [ 1533.781567] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1534.220074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1534.223256] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1534.258524] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1535.560745] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1539.497634] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1540.612467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1540.617118] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1540.653260] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1541.445796] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1545.740505] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1546.875356] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1659.412081] Lustre: 127209:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 1659.415064] Lustre: 127209:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 1659.417223] Lustre: 127209:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 1659.420168] Lustre: 127209:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 1659.421821] Lustre: 127209:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 1659.423407] Lustre: 127209:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1731.827831] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 11:21:06 (1776871266) [ 1743.327779] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1743.329669] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1743.332370] Lustre: Skipped 2 previous similar messages [ 1743.334346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1743.336037] Lustre: Skipped 7 previous similar messages [ 1743.404464] Lustre: server umount lustre-MDT0000 complete [ 1744.581188] LustreError: 127226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776871280 with bad export cookie 10651567307449776113 [ 1744.585160] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1745.377921] LustreError: 121649:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1745.382808] LustreError: 121649:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 1750.816175] Lustre: server umount lustre-MDT0001 complete [ 1766.367130] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1766.403165] Lustre: server umount lustre-OST0000 complete [ 1773.714932] Lustre: server umount lustre-OST0001 complete [ 1775.771710] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 1779.207585] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 1791.271387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1791.383701] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1791.397402] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1791.399444] Lustre: Skipped 1 previous similar message [ 1791.443623] Lustre: lustre-MDT0000: new disk, initializing [ 1791.445008] Lustre: Skipped 1 previous similar message [ 1791.468634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1791.472251] Lustre: Skipped 3 previous similar messages [ 1791.479793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1792.771228] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1796.902235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1796.948591] Lustre: 130185:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1797.052133] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1798.327651] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1801.375954] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1802.623901] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1804.979071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1806.263143] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1806.298833] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1807.015508] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 1810.285845] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1813.510395] Lustre: lustre-MDD0000: changelog on [ 1814.431467] Lustre: lustre-MDD0001: changelog on [ 2142.206094] Lustre: lustre-MDD0001: changelog off [ 2142.876238] Lustre: 130192:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 2143.215773] Lustre: lustre-MDD0000: changelog off [ 2149.346378] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2149.346425] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2149.349787] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2149.353456] Lustre: Skipped 5 previous similar messages [ 2149.360597] Lustre: Skipped 5 previous similar messages [ 2159.071132] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2159.138740] Lustre: server umount lustre-OST0000 complete [ 2166.586335] Lustre: server umount lustre-MDT0000 complete [ 2167.822281] LustreError: 130177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776871703 with bad export cookie 10651567307450580126 [ 2167.825059] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2167.825861] LustreError: 130177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2167.957615] Lustre: server umount lustre-MDT0001 complete [ 2171.665601] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 2172.680676] Key type lgssc unregistered [ 2172.808354] LNet: 140149:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2172.810546] LNetError: 140149:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2172.819285] LNet: Removed LNI 192.168.202.146@tcp [ 2173.130107] Key type .llcrypt unregistered [ 2173.131577] Key type ._llcrypt unregistered [ 2179.777811] Key type ._llcrypt registered [ 2179.778801] Key type .llcrypt registered [ 2179.816263] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 2186.084680] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 2186.529698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2186.537235] alg: No test for adler32 (adler32-zlib) [ 2187.387847] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 2187.473908] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 2189.055133] Key type lgssc registered [ 2189.438487] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2201.022382] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2201.027976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2202.119893] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2202.133842] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2202.170382] Lustre: lustre-MDT0000: new disk, initializing [ 2202.196368] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2202.204573] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2203.498436] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2207.721184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2207.770261] Lustre: 143288:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2207.793598] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2207.795802] Lustre: Skipped 1 previous similar message [ 2207.836584] Lustre: lustre-MDT0001: new disk, initializing [ 2207.867544] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2207.879245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2207.883358] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2209.079445] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2212.089737] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2213.244542] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2215.313435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2215.403959] Lustre: lustre-OST0000: new disk, initializing [ 2215.405463] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2215.430874] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2216.886775] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2216.890553] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2216.921959] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2217.114451] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2220.111672] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2222.307788] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_module obdecho/obdecho [ 2222.885799] LustreError: 145903:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 2222.888817] LustreError: 145903:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 2223.449431] LustreError: 145999:0:(echo_client.c:1882:echo_md_handler()) Can not resolve the path : rc = -2 [ 2227.169273] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2227.170172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2227.172576] Lustre: Skipped 1 previous similar message [ 2227.175684] Lustre: Skipped 2 previous similar messages [ 2232.288300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2232.290704] Lustre: Skipped 2 previous similar messages [ 2232.724489] Lustre: server umount lustre-MDT0000 complete [ 2233.906222] LustreError: 146253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776871769 with bad export cookie 7838564514576717198 [ 2233.909056] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2233.910552] LustreError: 146253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2234.012549] Lustre: server umount lustre-MDT0001 complete [ 2245.463159] Lustre: server umount lustre-OST0000 complete [ 2251.896413] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 2255.257468] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 2266.088842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2266.169670] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2266.178584] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2266.210755] Lustre: lustre-MDT0000: new disk, initializing [ 2266.231801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2266.237626] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2267.406927] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2271.289442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2271.324247] Lustre: 149677:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2271.342803] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2271.345258] Lustre: Skipped 1 previous similar message [ 2271.385091] Lustre: lustre-MDT0001: new disk, initializing [ 2271.414882] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2271.424738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2271.427052] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2272.587380] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2275.590230] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2276.713146] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2278.717126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2280.479205] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2283.325755] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2284.530384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2284.532701] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2284.560826] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2284.636683] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2284.703452] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2285.841777] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2285.903789] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2289.633705] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2289.634208] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2289.636159] LustreError: Skipped 1 previous similar message [ 2289.639903] Lustre: Skipped 1 previous similar message [ 2289.640395] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2289.644177] Lustre: Skipped 1 previous similar message [ 2292.564367] Lustre: server umount lustre-OST0000 complete [ 2294.752429] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2294.752907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2294.755198] Lustre: Skipped 2 previous similar messages [ 2294.757926] Lustre: Skipped 1 previous similar message [ 2299.872394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2299.874221] Lustre: Skipped 1 previous similar message [ 2300.061975] Lustre: server umount lustre-MDT0000 complete [ 2301.282902] LustreError: 149668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776871837 with bad export cookie 7838564514576718143 [ 2301.285773] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2301.286844] LustreError: 149668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2301.396602] Lustre: server umount lustre-MDT0001 complete [ 2302.493944] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 11:30:37 (1776871837) [ 2302.990052] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 2303.526894] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 11:30:38 (1776871838) [ 2305.939447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2306.095840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2306.097604] Lustre: Skipped 1 previous similar message [ 2307.487553] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2310.350370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2311.635337] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2313.289275] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2314.389516] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2316.425675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2318.238085] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2320.029439] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2330.449168] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.46@tcp (stopping) [ 2331.104432] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2331.106678] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2331.112567] Lustre: Skipped 2 previous similar messages [ 2331.216531] Lustre: server umount lustre-MDT0000 complete [ 2332.396510] LustreError: 153656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776871868 with bad export cookie 7838564514576718997 [ 2332.399470] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2332.399975] LustreError: 153656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2335.562740] LustreError: 155823:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2335.569890] LustreError: 155823:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 2336.223914] LustreError: 155134:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2336.224161] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2337.247830] LustreError: 153671:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2338.549269] Lustre: server umount lustre-MDT0001 complete [ 2340.764084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2344.899800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2344.978460] LustreError: 157171:0:(llog.c:1646:llog_backup()) MGC192.168.202.146@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 2344.981220] Lustre: 157171:0:(mgc_request_server.c:768:mgc_llog_local_copy()) MGC192.168.202.146@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 2356.042555] LustreError: 157176:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2361.162787] LustreError: 157176:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2361.167482] LustreError: 157176:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 2372.426647] LustreError: 157178:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2372.432014] LustreError: 157178:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 8 previous similar messages [ 2391.007728] LustreError: 157197:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2391.012093] LustreError: 157197:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 15 previous similar messages [ 2405.983246] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 2405.985322] Lustre: 157171:0:(mgc_request.c:1897:mgc_process_log()) MGC192.168.202.146@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 2426.698497] LustreError: 157176:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2426.704645] LustreError: 157176:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 21 previous similar messages [ 2446.822105] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2446.823982] Lustre: Skipped 2 previous similar messages [ 2448.063074] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2450.764969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2451.938471] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2452.831066] LustreError: 157897:0:(import.c:337:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 2452.833683] LustreError: 157897:0:(import.c:361:ptlrpc_invalidate_import()) @@@ still on sending list req@00000000e0824901 x1863185043907200/t0(0) o250->MGC192.168.202.146@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1776871988 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2452.839632] LustreError: 157897:0:(import.c:371:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 2452.959676] LustreError: 141514:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a372a80a x1863185043911424/t0(0) o250->MGC192.168.202.146@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2454.172504] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2455.758105] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2458.087911] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2458.324852] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2459.461493] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2463.043362] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 3 sec [ 2468.321610] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2468.322269] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2468.322470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2468.322474] Lustre: Skipped 6 previous similar messages [ 2468.324553] Lustre: Skipped 1 previous similar message [ 2469.519527] Lustre: server umount lustre-MDT0000 complete [ 2470.684163] LustreError: 153657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776872006 with bad export cookie 7838564514576725437 [ 2470.687050] LustreError: 153657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2470.687612] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2470.791944] Lustre: server umount lustre-MDT0001 complete [ 2481.181778] Lustre: server umount lustre-OST0000 complete [ 2488.266834] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 2491.792899] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 2504.017912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2504.104705] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2504.116393] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2504.118436] Lustre: Skipped 1 previous similar message [ 2504.154514] Lustre: lustre-MDT0000: new disk, initializing [ 2504.156732] Lustre: Skipped 1 previous similar message [ 2504.180249] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2504.182578] Lustre: Skipped 1 previous similar message [ 2504.188202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2505.517590] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2509.612869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2509.649341] Lustre: 161984:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2509.732791] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2510.877665] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2513.788598] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2515.042463] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2517.264270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2518.323788] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2518.352321] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2519.152586] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2522.170237] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2523.432645] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2523.504093] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2524.748401] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2524.812993] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2528.736867] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2528.738392] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2528.738990] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2528.738994] Lustre: Skipped 2 previous similar messages [ 2528.745118] Lustre: Skipped 2 previous similar messages [ 2531.529372] Lustre: server umount lustre-OST0000 complete [ 2538.832837] Lustre: server umount lustre-MDT0000 complete [ 2538.975829] LustreError: 161996:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2538.980945] LustreError: 161996:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 23 previous similar messages [ 2539.948114] LustreError: 161975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776872075 with bad export cookie 7838564514576742237 [ 2539.950816] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2539.951512] LustreError: 161975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2540.042242] Lustre: server umount lustre-MDT0001 complete [ 2542.624367] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 11:34:37 (1776872077) [ 2544.669532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2545.995332] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2548.679203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2549.986038] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2551.652292] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2552.795679] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2554.801546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2556.560146] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2558.205656] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2566.872613] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 11:35:02 (1776872102) [ 2569.185768] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2569.185806] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2569.188204] LustreError: Skipped 2 previous similar messages [ 2569.191835] Lustre: Skipped 2 previous similar messages [ 2573.840092] Lustre: server umount lustre-OST0000 complete [ 2579.424819] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2581.207768] Lustre: server umount lustre-MDT0000 complete [ 2582.467099] LustreError: 165968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776872118 with bad export cookie 7838564514576743105 [ 2582.469995] LustreError: 165968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2582.470393] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2582.580284] Lustre: server umount lustre-MDT0001 complete [ 2586.321207] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 2587.361859] Key type lgssc unregistered [ 2587.492380] LNet: 169354:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2587.494733] LNetError: 169354:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2587.505329] LNet: Removed LNI 192.168.202.146@tcp [ 2587.798779] Key type .llcrypt unregistered [ 2587.799868] Key type ._llcrypt unregistered [ 2595.636878] Key type ._llcrypt registered [ 2595.637925] Key type .llcrypt registered [ 2595.677611] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 2596.111285] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2596.124339] alg: No test for adler32 (adler32-zlib) [ 2596.983794] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 2597.066334] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 2598.647166] Key type lgssc registered [ 2599.010367] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2601.603212] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2601.611963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2617.055299] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2622.175218] LustreError: 170557:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.146@tcp: failed processing log, type 4: rc = -110 [ 2627.295338] LustreError: 170557:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 2627.300715] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 2627.302661] LustreError: MGC192.168.202.146@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 2627.307357] LustreError: 170557:0:(tgt_mount.c:1838:server_start_targets()) failed to start server lustre-OST0000: -22 [ 2627.310721] LustreError: 170557:0:(tgt_mount.c:2564:server_fill_super()) Unable to start targets: -22 [ 2627.313748] LustreError: 170557:0:(tgt_mount.c:2082:server_put_super()) no obd lustre-OST0000 [ 2627.338663] Lustre: server umount lustre-OST0000 complete [ 2627.340670] LustreError: 170557:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 2630.339061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2631.509729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2632.922388] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2635.968947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2636.092382] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2637.451775] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2639.421755] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2640.863407] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2643.477766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2643.551947] LustreError: 172644:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 2643.556924] Lustre: 172644:0:(mgc_request_server.c:748:mgc_llog_local_copy()) MGC192.168.202.146@tcp: invalid local config log lustre-OST0000: rc = -22 [ 2643.603372] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2646.320576] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2649.892624] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2651.617487] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2651.624475] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2651.635632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2652.131692] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2652.142878] Lustre: Skipped 1 previous similar message [ 2652.147592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2652.151101] Lustre: Skipped 2 previous similar messages [ 2657.251727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2657.257609] Lustre: Skipped 2 previous similar messages [ 2657.494911] Lustre: server umount lustre-MDT0000 complete [ 2660.019852] LustreError: 171038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776872195 with bad export cookie 16773031946249997728 [ 2660.024108] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2660.024961] LustreError: 171038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2660.194219] Lustre: server umount lustre-MDT0001 complete [ 2672.702390] Lustre: server umount lustre-OST0000 complete [ 2678.922616] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 2686.364977] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 2715.537896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2715.686422] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2715.709628] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2715.755508] Lustre: lustre-MDT0000: new disk, initializing [ 2715.809771] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2715.820353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2718.019057] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2724.138712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2724.200827] Lustre: 176015:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2724.227541] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2724.231136] Lustre: Skipped 1 previous similar message [ 2724.278484] Lustre: lustre-MDT0001: new disk, initializing [ 2724.328868] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2724.361303] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2724.367496] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2726.517524] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2730.623677] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2732.417901] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2736.148849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2736.330462] Lustre: lustre-OST0000: new disk, initializing [ 2736.333879] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2736.376831] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2738.956163] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2742.282303] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2742.288107] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2742.342740] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2742.771446] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2744.813251] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2744.933558] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2747.219650] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2747.317838] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2752.484380] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2752.485273] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2752.488113] LustreError: Skipped 1 previous similar message [ 2752.495088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2752.499769] Lustre: Skipped 1 previous similar message [ 2754.392869] Lustre: server umount lustre-OST0000 complete [ 2757.600842] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2757.603177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2757.606549] Lustre: Skipped 2 previous similar messages [ 2762.411208] Lustre: server umount lustre-MDT0000 complete [ 2762.722748] LustreError: 176028:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2762.728333] LustreError: 176028:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 2764.407106] LustreError: 176009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776872300 with bad export cookie 16773031946249998428 [ 2764.409210] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2764.412215] LustreError: 176009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2764.589140] Lustre: server umount lustre-MDT0001 complete [ 2769.118774] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 11:38:24 (1776872304) [ 2776.946296] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 2778.185540] Key type lgssc unregistered [ 2778.334776] LNet: 180532:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2778.341351] LNetError: 180532:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2778.349752] LNet: Removed LNI 192.168.202.146@tcp [ 2778.664707] Key type .llcrypt unregistered [ 2778.666386] Key type ._llcrypt unregistered [ 2787.774616] Key type ._llcrypt registered [ 2787.776531] Key type .llcrypt registered [ 2787.843780] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 2788.341049] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2788.360470] alg: No test for adler32 (adler32-zlib) [ 2789.242141] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 2789.366769] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 2790.978090] Key type lgssc registered [ 2791.471894] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2794.919496] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2794.931648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2796.114740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2797.662313] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2800.948650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2801.085335] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2802.777709] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2805.342178] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2807.291295] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2811.792621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2811.909515] Lustre: *** cfs_fail_loc=131e, val=0*** [ 2811.911581] LustreError: 183346:0:(llog.c:1687:llog_backup()) MGC192.168.202.146@tcp: failed to backup log lustre-OST0000: rc = -28 [ 2811.916293] Lustre: 183346:0:(mgc_request_server.c:745:mgc_llog_local_copy()) MGC192.168.202.146@tcp: can't backup local config lustre-OST0000: rc = -28 [ 2812.009753] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2815.375972] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2818.092570] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2821.600343] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2821.605509] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2821.610599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2822.113716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2822.116049] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2822.118075] Lustre: Skipped 1 previous similar message [ 2822.125695] Lustre: Skipped 1 previous similar message [ 2825.188093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2825.194409] Lustre: Skipped 3 previous similar messages [ 2825.318715] Lustre: server umount lustre-MDT0000 complete [ 2827.157140] LustreError: 181690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776872363 with bad export cookie 9968872835124674677 [ 2827.160333] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2827.163396] LustreError: 181690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2827.231853] LustreError: 182424:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000aedd5a98 x1863185675078400/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2827.309210] Lustre: server umount lustre-MDT0001 complete [ 2839.408927] Lustre: server umount lustre-OST0000 complete [ 2844.024454] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 2849.143869] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 2868.168926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2868.304211] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2868.325235] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2868.370965] Lustre: lustre-MDT0000: new disk, initializing [ 2868.434818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2868.446770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2870.610303] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2876.376040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2876.438211] Lustre: 186716:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2876.466188] Lustre: *** cfs_fail_loc=131e, val=0*** [ 2876.467693] Lustre: Skipped 1 previous similar message [ 2876.469079] LustreError: 187481:0:(llog.c:1687:llog_backup()) MGC192.168.202.146@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 2876.472042] LustreError: 187481:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 2876.475876] Lustre: 187481:0:(mgc_request_server.c:768:mgc_llog_local_copy()) MGC192.168.202.146@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 2876.486716] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2876.490092] Lustre: Skipped 1 previous similar message [ 2876.530975] Lustre: lustre-MDT0001: new disk, initializing [ 2876.591228] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2876.611841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2876.624699] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2878.774585] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2883.002905] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2884.973325] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2888.724732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2888.863688] Lustre: *** cfs_fail_loc=131e, val=0*** [ 2888.864876] Lustre: Skipped 1 previous similar message [ 2888.869251] LustreError: 188432:0:(llog.c:1687:llog_backup()) MGC192.168.202.146@tcp: failed to backup log lustre-OST0000: rc = -28 [ 2888.872758] LustreError: 188432:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 2888.876366] Lustre: 188432:0:(mgc_request_server.c:768:mgc_llog_local_copy()) MGC192.168.202.146@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 2888.879598] Lustre: 188432:0:(mgc_request_server.c:768:mgc_llog_local_copy()) Skipped 1 previous similar message [ 2888.890311] Lustre: lustre-OST0000: new disk, initializing [ 2888.893253] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2888.945705] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2889.423589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2889.433087] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2889.478576] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2892.370939] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2897.189627] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2899.453665] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2899.553444] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2901.708899] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2901.862127] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2905.057525] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2905.058349] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2905.060817] LustreError: Skipped 1 previous similar message [ 2905.062596] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2905.067796] Lustre: Skipped 1 previous similar message [ 2910.179332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2910.182148] Lustre: Skipped 2 previous similar messages [ 2917.343364] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2917.424241] Lustre: server umount lustre-OST0000 complete [ 2920.417235] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2920.419514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2920.422829] Lustre: Skipped 1 previous similar message [ 2920.425489] Lustre: Skipped 2 previous similar messages [ 2925.508449] Lustre: server umount lustre-MDT0000 complete [ 2925.536096] LustreError: 186729:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2925.543860] LustreError: 186729:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 2927.265082] LustreError: 188455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776872463 with bad export cookie 9968872835124675370 [ 2927.268100] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2927.269959] LustreError: 188455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2927.438332] Lustre: server umount lustre-MDT0001 complete [ 2932.504843] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 11:41:07 (1776872467) [ 2940.840579] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 2945.585160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2945.784023] LustreError: 191419:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2945.810927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2947.707460] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2951.141641] LustreError: 191420:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2954.972847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2958.728885] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2961.350455] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2967.440951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2971.102837] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2976.087592] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2976.188461] Lustre: lustre-OST0001: new disk, initializing [ 2976.190596] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2978.396573] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2978.400763] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2978.469525] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2979.397182] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 2986.593370] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2990.209409] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2996.382979] Lustre: DEBUG MARKER: ADD OST3 [ 2999.459410] loop: module loaded [ 2999.495314] loop0: detected capacity change from 0 to 204800000 [ 2999.622676] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 2999.626840] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 2999.633658] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3002.959961] loop0: detected capacity change from 0 to 204800000 [ 3005.448333] Lustre: DEBUG MARKER: STOP OST3 [ 3006.459430] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3006.564555] Lustre: lustre-OST0003: new disk, initializing [ 3006.574035] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 3006.641241] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 3006.643333] Lustre: Skipped 3 previous similar messages [ 3008.238158] Lustre: *** cfs_fail_loc=2109, val=2*** [ 3008.322750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 3008.329908] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 3008.371725] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 3009.050986] Lustre: Failing over lustre-OST0003 [ 3009.122776] Lustre: server umount lustre-OST0003 complete [ 3010.273565] LustreError: 195747:0:(osp_precreate.c:1280:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 3010.385784] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3013.477860] LustreError: 193281:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.202.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3013.599880] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3013.605702] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3013.612165] Lustre: Skipped 1 previous similar message [ 3016.830508] Lustre: DEBUG MARKER: START OST3 again [ 3018.581242] LustreError: 193281:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.202.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3018.596523] LustreError: 193281:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 3021.211068] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3021.373137] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 3023.333881] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3023.425306] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 3023.432946] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3025.126787] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3041.730770] Lustre: Failing over lustre-OST0003 [ 3041.767571] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3041.773500] Lustre: lustre-OST0003: Not available for connect from 0@lo (stopping) [ 3041.775274] Lustre: Skipped 1 previous similar message [ 3041.817264] Lustre: server umount lustre-OST0003 complete [ 3046.887896] LustreError: 192870:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3046.915402] LustreError: 192870:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 3062.410709] Lustre: server umount lustre-MDT0000 complete [ 3064.358440] LustreError: 191400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776872600 with bad export cookie 9968872835124676259 [ 3064.364087] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3064.373817] LustreError: 191400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3067.363138] LustreError: 191416:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3067.365657] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3067.372513] LustreError: 191416:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 5 previous similar messages [ 3067.390553] Lustre: Skipped 6 previous similar messages [ 3070.698886] Lustre: server umount lustre-MDT0001 complete [ 3073.303302] Lustre: server umount lustre-OST0000 complete [ 3075.542210] Lustre: server umount lustre-OST0001 complete [ 3078.615348] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 3082.928707] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 3103.352785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3103.488143] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3103.505687] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3103.570328] Lustre: lustre-MDT0000: new disk, initializing [ 3103.635422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3103.640243] Lustre: Skipped 1 previous similar message [ 3103.650089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3105.844821] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3112.057545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3112.122273] Lustre: 200884:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3112.282323] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3114.487956] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3118.851536] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3120.863377] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3124.644079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3124.787888] Lustre: lustre-OST0000: new disk, initializing [ 3124.789355] Lustre: Skipped 1 previous similar message [ 3124.793119] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3124.796723] Lustre: Skipped 2 previous similar messages [ 3126.040948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3126.045739] Lustre: Skipped 1 previous similar message [ 3126.049262] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3126.102806] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3128.168688] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3132.562702] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3134.883764] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3134.987808] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3137.062289] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3137.176185] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3141.606838] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3141.607212] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3141.608057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3141.608062] Lustre: Skipped 12 previous similar messages [ 3141.611899] LustreError: Skipped 1 previous similar message [ 3141.617068] Lustre: Skipped 1 previous similar message [ 3152.351135] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3152.430577] Lustre: server umount lustre-OST0000 complete [ 3160.552723] Lustre: server umount lustre-MDT0000 complete [ 3162.080951] LustreError: 200892:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3162.093126] LustreError: 200892:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 3162.436572] LustreError: 200875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776872698 with bad export cookie 9968872835124678555 [ 3162.438318] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3162.443124] LustreError: 200875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3162.619636] Lustre: server umount lustre-MDT0001 complete [ 3164.629730] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 11:44:59 (1776872699) [ 3173.828233] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 3179.353325] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 3199.074877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3199.198845] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3199.216115] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3199.260426] Lustre: lustre-MDT0000: new disk, initializing [ 3199.314642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3201.521647] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3207.737623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3207.796297] Lustre: 207064:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3207.926618] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3209.937561] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3214.965453] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3218.032991] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3225.806175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3227.395278] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3233.504693] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3240.663957] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3245.868404] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3246.113705] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3251.439655] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3251.686969] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3258.353769] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3258.378081] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3258.379956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3258.379966] Lustre: Skipped 7 previous similar messages [ 3258.394311] Lustre: Skipped 3 previous similar messages [ 3260.159291] Lustre: server umount lustre-OST0000 complete [ 3269.854134] Lustre: server umount lustre-MDT0000 complete [ 3272.853769] LustreError: 207055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776872808 with bad export cookie 9968872835124679430 [ 3272.858541] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3272.862305] LustreError: 207055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3273.156660] Lustre: server umount lustre-MDT0001 complete [ 3280.023837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3280.341304] LustreError: 210872:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3280.383292] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3280.387900] Lustre: Skipped 5 previous similar messages [ 3283.613525] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3289.565681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3292.919804] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3297.238390] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3299.845222] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3304.258047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3309.166256] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3313.273878] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3327.755945] Lustre: server umount lustre-OST0000 complete [ 3336.783973] Lustre: server umount lustre-MDT0000 complete [ 3339.518799] LustreError: 211592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776872875 with bad export cookie 9968872835124680312 [ 3339.521389] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3339.526340] LustreError: 211592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3339.730125] Lustre: server umount lustre-MDT0001 complete [ 3451.610324] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 3453.685918] Key type lgssc unregistered [ 3453.904879] LNet: 214177:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3453.912594] LNetError: 214177:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3453.934135] LNet: Removed LNI 192.168.202.146@tcp [ 3454.520519] Key type .llcrypt unregistered [ 3454.525536] Key type ._llcrypt unregistered [ 3473.355428] Key type ._llcrypt registered [ 3473.357020] Key type .llcrypt registered [ 3473.436774] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 3485.394133] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 3486.198821] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3486.211283] alg: No test for adler32 (adler32-zlib) [ 3487.204718] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 3487.372469] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 3489.055142] Key type lgssc registered [ 3489.930464] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3516.383082] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3516.408656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3517.620543] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3517.649322] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3517.703550] Lustre: lustre-MDT0000: new disk, initializing [ 3517.759204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3517.779458] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3520.682394] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3527.947427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3528.019092] Lustre: 217474:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3528.062200] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3528.065735] Lustre: Skipped 1 previous similar message [ 3528.124371] Lustre: lustre-MDT0001: new disk, initializing [ 3528.190946] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3528.225470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3528.233199] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3531.218536] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3536.975784] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3539.665665] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3544.727965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3544.884421] Lustre: lustre-OST0000: new disk, initializing [ 3544.888029] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3544.969188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3549.419815] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3552.799342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3552.809163] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3552.896714] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3555.417376] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3558.717787] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3558.850459] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3561.816736] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3561.983353] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3563.490383] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3563.490414] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3563.497395] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3563.515561] Lustre: Skipped 1 previous similar message [ 3568.611454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3568.616828] Lustre: Skipped 1 previous similar message [ 3569.538794] Lustre: server umount lustre-OST0000 complete [ 3573.729486] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3573.744941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3573.751240] Lustre: Skipped 2 previous similar messages [ 3578.289550] Lustre: server umount lustre-MDT0000 complete [ 3580.879901] LustreError: 217465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776873116 with bad export cookie 9228228046253197748 [ 3580.880419] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3580.886081] LustreError: 217465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3581.146925] Lustre: server umount lustre-MDT0001 complete [ 3594.390609] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 3601.618624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3601.937352] LustreError: 221980:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3601.955338] LustreError: 221980:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 3601.999851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3604.982335] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3607.010093] LustreError: 221981:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3611.106521] LustreError: 221980:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3611.377579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3611.636439] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3614.814689] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3617.095858] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3622.397340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3622.816304] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3627.339384] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3634.253369] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3634.362641] Lustre: lustre-OST0001: new disk, initializing [ 3634.366097] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3638.402614] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3643.500120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3643.508769] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3643.600471] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3647.911627] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3659.286328] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3674.344720] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 11:53:28 (1776873208) [ 3678.178889] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3678.183906] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3678.197420] Lustre: Skipped 1 previous similar message [ 3678.206609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3681.976380] Lustre: server umount lustre-MDT0000 complete [ 3684.621596] LustreError: 221962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776873220 with bad export cookie 9228228046253198630 [ 3684.624650] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3684.628845] LustreError: 221962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3684.860569] Lustre: server umount lustre-MDT0001 complete [ 3697.750926] Lustre: server umount lustre-OST0000 complete [ 3710.739933] Lustre: server umount lustre-OST0001 complete [ 3714.940426] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 3721.921974] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 3744.429108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3744.671874] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3744.707160] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3744.774638] Lustre: lustre-MDT0000: new disk, initializing [ 3744.851822] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3744.854582] Lustre: Skipped 1 previous similar message [ 3744.868845] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3748.177480] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3755.393424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3755.471842] Lustre: 228160:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3755.516051] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3755.522784] Lustre: Skipped 1 previous similar message [ 3755.582873] Lustre: lustre-MDT0001: new disk, initializing [ 3755.672205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3755.680275] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3758.273279] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3763.170756] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3765.467677] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3769.762381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3771.302982] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3771.372191] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3774.148337] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3779.279894] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3782.708687] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3782.872541] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3785.979988] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3786.133788] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3791.840144] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3791.840462] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3791.847574] LustreError: Skipped 1 previous similar message [ 3791.848793] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3791.856056] Lustre: Skipped 4 previous similar messages [ 3791.858917] Lustre: Skipped 4 previous similar messages [ 3793.711408] Lustre: server umount lustre-OST0000 complete [ 3796.960643] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3802.083908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3802.088570] Lustre: Skipped 4 previous similar messages [ 3802.533239] Lustre: server umount lustre-MDT0000 complete [ 3805.026325] LustreError: 228152:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776873340 with bad export cookie 9228228046253200772 [ 3805.028807] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3805.036221] LustreError: 228152:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3805.227037] Lustre: server umount lustre-MDT0001 complete [ 3810.651504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3810.944207] LustreError: 231971:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3810.981594] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3810.986047] Lustre: Skipped 2 previous similar messages [ 3813.957551] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3816.421243] LustreError: 231972:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3819.845934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3822.733122] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3826.839149] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3829.787353] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3834.841396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3839.112884] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 3842.638603] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3989.985581] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3989.990303] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3990.002230] Lustre: Skipped 1 previous similar message [ 3990.022695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3994.369194] Lustre: server umount lustre-OST0000 complete [ 4003.302288] LustreError: 231972:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4003.313929] LustreError: 231972:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 4003.347897] Lustre: server umount lustre-MDT0000 complete [ 4005.913393] LustreError: 232691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776873541 with bad export cookie 9228228046253201654 [ 4005.917338] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4005.922200] LustreError: 232691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4006.155655] Lustre: server umount lustre-MDT0001 complete [ 4110.291935] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 4111.744693] Key type lgssc unregistered [ 4111.921277] LNet: 235218:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4111.925166] LNetError: 235218:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4111.938860] LNet: Removed LNI 192.168.202.146@tcp [ 4112.416229] Key type .llcrypt unregistered [ 4112.418431] Key type ._llcrypt unregistered [ 4126.440612] Key type ._llcrypt registered [ 4126.441976] Key type .llcrypt registered [ 4126.517870] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 4134.729957] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 4135.144798] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4135.175828] alg: No test for adler32 (adler32-zlib) [ 4136.079558] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 4136.209899] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 4137.815181] Key type lgssc registered [ 4138.283415] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4161.571648] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 4167.547058] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4167.560333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4168.736216] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4168.754443] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4168.802634] Lustre: lustre-MDT0000: new disk, initializing [ 4168.844328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4168.855616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4171.207590] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4177.021657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4177.084750] Lustre: 239222:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4177.134135] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4177.138129] Lustre: Skipped 1 previous similar message [ 4177.188712] Lustre: lustre-MDT0001: new disk, initializing [ 4177.232831] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4177.253191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4177.257068] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4179.129840] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4181.919734] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4184.924664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4185.067178] Lustre: lustre-OST0000: new disk, initializing [ 4185.070631] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4185.118098] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4188.099796] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4192.274402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4192.284328] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4192.366300] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4193.610812] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4193.687345] Lustre: lustre-OST0001: new disk, initializing [ 4193.690408] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4193.748153] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4196.554773] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4198.438118] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4198.444625] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4198.489069] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4204.811556] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4209.514322] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4221.370284] Lustre: server umount lustre-MDT0000 complete [ 4222.884222] LustreError: 240000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776873758 with bad export cookie 6958291042944520937 [ 4222.889599] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4222.894778] LustreError: 240000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4223.970536] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4223.975434] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4223.980253] Lustre: Skipped 2 previous similar messages [ 4223.986916] Lustre: Skipped 1 previous similar message [ 4225.107331] Lustre: server umount lustre-MDT0001 complete [ 4233.154141] Lustre: server umount lustre-OST0000 complete [ 4234.875763] Lustre: server umount lustre-OST0001 complete [ 4241.307870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4245.495374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4249.001510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4253.237784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4261.277700] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 4265.796783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4265.806571] Lustre: lustre-MDT0000: reset Object Index mappings [ 4265.988306] LustreError: 246637:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4266.001279] LustreError: 246637:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 3 previous similar messages [ 4266.036578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4267.739434] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4271.077509] LustreError: 246638:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4271.085183] LustreError: 246638:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 4271.513233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4271.523713] Lustre: lustre-MDT0001: reset Object Index mappings [ 4273.302724] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4274.532966] Lustre: 247743:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4277.326273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4277.492349] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4277.495055] Lustre: Skipped 1 previous similar message [ 4280.031589] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4281.575467] LustreError: 248098:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4283.653316] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4286.403408] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4291.056324] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 4291.059971] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4295.218531] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4296.809205] Lustre: 249579:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4306.402464] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4306.404076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4306.406404] Lustre: Skipped 3 previous similar messages [ 4306.408873] Lustre: Skipped 1 previous similar message [ 4309.486042] Lustre: server umount lustre-MDT0000 complete [ 4310.991730] LustreError: 246620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776873846 with bad export cookie 6958291042944527727 [ 4310.993539] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4310.997310] LustreError: 246620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4311.134618] Lustre: server umount lustre-MDT0001 complete [ 4322.955661] Lustre: server umount lustre-OST0000 complete [ 4334.754449] Lustre: server umount lustre-OST0001 complete [ 4343.172220] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 12:04:38 (1776873878) [ 4349.860159] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 4353.931430] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 4368.778781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4368.875715] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4368.888930] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4368.927940] Lustre: lustre-MDT0000: new disk, initializing [ 4368.954755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4368.957421] Lustre: Skipped 1 previous similar message [ 4368.964554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4370.439594] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4374.902588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4374.948178] Lustre: 253796:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4375.060342] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4376.558196] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4379.965277] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4381.351259] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4383.837442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4383.944470] Lustre: lustre-OST0000: new disk, initializing [ 4383.945969] Lustre: Skipped 1 previous similar message [ 4383.948265] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4383.951160] Lustre: Skipped 2 previous similar messages [ 4385.477175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4385.480560] Lustre: Skipped 1 previous similar message [ 4385.482443] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4385.526419] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4386.124940] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4389.601766] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4391.195874] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4391.276327] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4392.849625] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4392.936804] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4396.002153] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4396.002481] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4396.003140] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4396.003144] Lustre: Skipped 2 previous similar messages [ 4396.006151] Lustre: Skipped 1 previous similar message [ 4401.122310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4401.124893] Lustre: Skipped 2 previous similar messages [ 4406.242614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4406.245333] Lustre: Skipped 1 previous similar message [ 4408.287126] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4408.346442] Lustre: server umount lustre-OST0000 complete [ 4410.847849] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4410.850277] LustreError: Skipped 1 previous similar message [ 4410.851564] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4415.764754] Lustre: server umount lustre-MDT0000 complete [ 4416.480109] LustreError: 253808:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4416.485528] LustreError: 253808:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 4417.035473] LustreError: 253789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776873952 with bad export cookie 6958291042944529820 [ 4417.036152] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4417.041323] LustreError: 253789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4417.160691] Lustre: server umount lustre-MDT0001 complete [ 4423.309109] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 4427.370704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4427.539140] LustreError: 258302:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4427.564673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4427.569158] Lustre: Skipped 2 previous similar messages [ 4429.225679] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4433.058138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4434.990470] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4436.273313] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4439.271402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4441.837793] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4445.564581] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4445.659619] Lustre: lustre-OST0001: new disk, initializing [ 4445.662268] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4447.442174] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4447.447465] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4447.509693] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4448.294452] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4453.495123] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4455.006100] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4462.211888] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 4462.219635] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 4468.196265] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4468.199585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4468.200814] Lustre: Skipped 4 previous similar messages [ 4468.204810] Lustre: Skipped 6 previous similar messages [ 4470.372938] Lustre: server umount lustre-MDT0000 complete [ 4471.713085] LustreError: 260558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776874007 with bad export cookie 6958291042944530702 [ 4471.716129] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4471.717662] LustreError: 260558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4471.857038] Lustre: server umount lustre-MDT0001 complete [ 4483.523059] Lustre: server umount lustre-OST0000 complete [ 4494.180760] Lustre: server umount lustre-OST0001 complete [ 4499.974143] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 4503.797720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4503.935903] LustreError: 263577:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4503.942886] LustreError: 263577:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 4503.964844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4503.966942] Lustre: Skipped 3 previous similar messages [ 4505.358280] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4508.662518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4510.446982] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4511.627978] Lustre: 264682:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4514.278331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4516.830592] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4520.053354] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4522.348919] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4528.618405] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 4528.623858] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 4536.609479] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4538.110550] Lustre: 266549:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4545.132048] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 4545.151132] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 4548.905733] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 12:08:03 (1776874083) [ 4564.450407] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4564.451440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4564.454888] Lustre: Skipped 3 previous similar messages [ 4564.457071] Lustre: Skipped 2 previous similar messages [ 4566.694293] Lustre: server umount lustre-MDT0000 complete [ 4568.105436] LustreError: 264685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776874104 with bad export cookie 6958291042944537009 [ 4568.106054] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4568.114295] LustreError: 264685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4569.567921] LustreError: 263573:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4569.577812] LustreError: 263573:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 4574.369307] Lustre: server umount lustre-MDT0001 complete [ 4582.153695] Lustre: server umount lustre-OST0000 complete [ 4589.654118] Lustre: server umount lustre-OST0001 complete [ 4591.704223] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 4595.239420] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 4610.743477] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 4614.027780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4614.112463] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4614.124840] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4614.162244] Lustre: lustre-MDT0000: new disk, initializing [ 4614.194062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4615.518657] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4619.644848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4619.686146] Lustre: 270733:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4619.785450] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4621.061507] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4625.459044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4626.137423] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4627.504686] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4631.643965] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4633.446126] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4633.737497] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4638.174509] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4648.270240] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 12:09:43 (1776874183) [ 4650.463858] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4650.466403] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4650.470527] Lustre: Skipped 2 previous similar messages [ 4650.472597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4650.474995] Lustre: Skipped 4 previous similar messages [ 4655.127393] Lustre: server umount lustre-MDT0000 complete [ 4656.290403] LustreError: 274201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776874192 with bad export cookie 6958291042944544877 [ 4656.292513] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4656.295127] LustreError: 274201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4656.398080] Lustre: server umount lustre-MDT0001 complete [ 4667.392552] Lustre: server umount lustre-OST0000 complete [ 4678.875101] Lustre: server umount lustre-OST0001 complete [ 4681.596319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4681.747892] LustreError: 275414:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4681.755046] LustreError: 275414:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 4681.776109] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4681.777934] Lustre: Skipped 7 previous similar messages [ 4682.992983] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4685.593053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4686.910973] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4688.605337] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4689.771549] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4691.964494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4693.812905] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4695.442439] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4717.665199] loop1: detected capacity change from 0 to 204800000 [ 4717.740106] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4717.744318] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4717.749627] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4719.509095] loop1: detected capacity change from 0 to 204800000 [ 4721.418929] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4721.446336] LustreError: 275402:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) fs3865: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 4721.451744] LustreError: 275402:0:(mgs_llog.c:731:mgs_check_index()) Can't get db for fs3865 [ 4721.454479] LustreError: 275402:0:(mgs_handler.c:517:mgs_target_reg()) fs3865-MDT0001: Error checking target index: rc = -13 [ 4721.458413] LustreError: fs3865-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 4721.462165] LustreError: 278862:0:(tgt_mount.c:2564:server_fill_super()) Unable to start targets: -13 [ 4721.464765] LustreError: 278862:0:(tgt_mount.c:2082:server_put_super()) no obd fs3865-MDT0001 [ 4721.467057] LustreError: 278862:0:(tgt_mount.c:114:server_deregister_mount()) fs3865-MDT0001 not registered [ 4721.478498] Lustre: server umount fs3865-MDT0001 complete [ 4721.479993] LustreError: 278862:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 4724.482744] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4724.511042] Lustre: Setting parameter fs3865-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs3865-MDT0001 [ 4724.514604] Lustre: Skipped 2 previous similar messages [ 4724.534553] Lustre: srv-fs3865-MDT0001: No data found on store. Initialize space. [ 4724.536217] Lustre: Skipped 4 previous similar messages [ 4724.575386] Lustre: fs3865-MDT0001: new disk, initializing [ 4724.577243] Lustre: Skipped 3 previous similar messages [ 4724.618715] Lustre: srv-fs3865-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 4724.621669] Lustre: 279329:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs3865-MDT0001-mdtlov: can't create new llog: rc = -115 [ 4724.624885] LustreError: 279329:0:(lod_dev.c:510:lod_sub_recovery_thread()) fs3865-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 4726.092229] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4729.063448] Lustre: server umount fs3865-MDT0001 complete [ 4733.588589] loop1: detected capacity change from 0 to 204800000 [ 4735.509388] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4735.628446] Lustre: 280876:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs3865-MDT0001-mdtlov: can't create new llog: rc = -115 [ 4735.632076] LustreError: 280876:0:(lod_dev.c:510:lod_sub_recovery_thread()) fs3865-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 4736.974141] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4738.611346] Lustre: server umount fs3865-MDT0001 complete [ 4741.969735] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.46@tcp (stopping) [ 4741.972878] Lustre: Skipped 4 previous similar messages [ 4742.111768] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4742.114174] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4742.118824] Lustre: Skipped 3 previous similar messages [ 4747.083838] LustreError: 277525:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4747.090303] LustreError: 277525:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 25 previous similar messages [ 4747.927936] Lustre: server umount lustre-MDT0000 complete [ 4750.852788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4750.891636] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4752.179543] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4754.569316] loop1: detected capacity change from 0 to 204800000 [ 4756.405506] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4756.454683] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4756.544178] Lustre: 283284:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs3865-MDT0001-mdtlov: can't create new llog: rc = -115 [ 4756.547389] LustreError: 283284:0:(lod_dev.c:510:lod_sub_recovery_thread()) fs3865-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 4757.837376] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4759.359390] Lustre: server umount fs3865-MDT0001 complete [ 4766.690664] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4772.443325] Lustre: server umount lustre-OST0000 complete [ 4776.929029] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4788.191113] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4788.246083] Lustre: server umount lustre-MDT0000 complete [ 4789.430083] LustreError: 275396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776874325 with bad export cookie 6958291042944547558 [ 4789.433588] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4789.434380] LustreError: 275396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4789.542804] Lustre: server umount lustre-MDT0001 complete [ 4793.087688] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 4794.190135] Key type lgssc unregistered [ 4794.322417] LNet: 285591:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4794.325923] LNetError: 285591:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4794.334304] LNet: Removed LNI 192.168.202.146@tcp [ 4794.654125] Key type .llcrypt unregistered [ 4794.655205] Key type ._llcrypt unregistered [ 4802.113290] Key type ._llcrypt registered [ 4802.114307] Key type .llcrypt registered [ 4802.148238] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 4802.453901] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4802.460573] alg: No test for adler32 (adler32-zlib) [ 4803.301216] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 4803.381158] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 4804.959189] Key type lgssc registered [ 4805.306137] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4808.064794] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4808.071515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4809.242076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4810.497751] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4813.346969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4813.472837] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4814.642922] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4816.355036] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4817.425825] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4819.573987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4819.690733] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4820.708238] LustreError: 288440:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4820.717132] LustreError: 288440:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 4821.652957] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4823.473135] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4823.776887] LustreError: 288439:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4823.781483] LustreError: 288439:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 1 previous similar message [ 4829.154418] LustreError: 288438:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4829.160791] LustreError: 288438:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 4830.981703] loop1: detected capacity change from 0 to 204800000 [ 4831.038032] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4831.043019] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4831.046833] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4833.090274] loop1: detected capacity change from 0 to 204800000 [ 4834.275972] LustreError: 288628:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4834.281322] LustreError: 288628:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 4835.089129] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4835.143196] Lustre: fs27521-OST0000: new disk, initializing [ 4835.145862] Lustre: srv-fs27521-OST0000: No data found on store. Initialize space. [ 4835.171046] Lustre: fs27521-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4837.123781] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4839.396261] LustreError: 288628:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4839.401218] LustreError: 288628:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 4840.657737] Lustre: server umount fs27521-OST0000 complete [ 4844.695527] loop1: detected capacity change from 0 to 204800000 [ 4846.569705] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4846.632374] Lustre: fs27521-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4848.500055] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4849.635474] LustreError: 288628:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4849.640362] LustreError: 288628:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 5 previous similar messages [ 4851.474532] loop2: detected capacity change from 0 to 204800000 [ 4851.567655] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4851.571211] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4851.575316] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4853.264892] loop2: detected capacity change from 0 to 204800000 [ 4855.004526] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4855.031816] LustreError: fs27521-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 4855.035825] LustreError: fs27521-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 4855.038321] LustreError: 293277:0:(tgt_mount.c:2564:server_fill_super()) Unable to start targets: -13 [ 4855.040573] LustreError: 293277:0:(tgt_mount.c:2082:server_put_super()) no obd fs27521-OST0001 [ 4855.042329] LustreError: 293277:0:(tgt_mount.c:114:server_deregister_mount()) fs27521-OST0001 not registered [ 4855.051197] Lustre: server umount fs27521-OST0001 complete [ 4855.052440] LustreError: 293277:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 4858.280216] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4858.321247] Lustre: fs27521-OST0001: new disk, initializing [ 4858.323235] Lustre: srv-fs27521-OST0001: No data found on store. Initialize space. [ 4858.346522] Lustre: fs27521-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4860.200496] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4863.594659] Lustre: server umount fs27521-OST0000 complete [ 4865.866831] LustreError: 288628:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4865.872229] LustreError: 288628:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 9 previous similar messages [ 4866.603593] Lustre: server umount fs27521-OST0001 complete [ 4870.381081] loop1: detected capacity change from 0 to 204800000 [ 4872.245451] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4874.215568] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4876.632835] loop2: detected capacity change from 0 to 204800000 [ 4878.532823] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4878.591872] Lustre: fs27521-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4878.594678] Lustre: Skipped 1 previous similar message [ 4880.466399] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4883.468502] Lustre: server umount fs27521-OST0000 complete [ 4886.393359] Lustre: server umount fs27521-OST0001 complete [ 4890.593615] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4890.594184] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4890.597556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4890.600878] Lustre: Skipped 1 previous similar message [ 4895.715148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4895.718335] Lustre: Skipped 1 previous similar message [ 4896.140693] Lustre: server umount lustre-OST0000 complete [ 4900.832829] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4900.832863] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4900.833928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4900.838134] Lustre: Skipped 2 previous similar messages [ 4903.520545] Lustre: server umount lustre-MDT0000 complete [ 4904.912197] LustreError: 286817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776874440 with bad export cookie 6820907547429216374 [ 4904.915489] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4904.916644] LustreError: 286817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4905.037242] Lustre: server umount lustre-MDT0001 complete [ 4908.538970] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 4909.524725] Key type lgssc unregistered [ 4909.660374] LNet: 299670:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4909.663649] LNetError: 299670:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4909.672327] LNet: Removed LNI 192.168.202.146@tcp [ 4909.962248] Key type .llcrypt unregistered [ 4909.963620] Key type ._llcrypt unregistered [ 4920.563805] Key type ._llcrypt registered [ 4920.564617] Key type .llcrypt registered [ 4920.605367] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 4927.484356] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 4927.855743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4927.912216] alg: No test for adler32 (adler32-zlib) [ 4928.770419] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 4928.854815] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 4930.431095] Key type lgssc registered [ 4930.800313] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4943.293216] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4943.298408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4944.390620] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4944.403777] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4944.441496] Lustre: lustre-MDT0000: new disk, initializing [ 4944.481026] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4944.489450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4945.896531] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4950.145527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4950.189340] Lustre: 303468:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4950.213660] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4950.215948] Lustre: Skipped 1 previous similar message [ 4950.249573] Lustre: lustre-MDT0001: new disk, initializing [ 4950.279732] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4950.292047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4950.295234] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4951.636807] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4954.816533] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4956.035732] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4958.243114] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4958.332914] Lustre: lustre-OST0000: new disk, initializing [ 4958.335573] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4958.361712] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4960.078672] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 4962.998333] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4966.393203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4966.397534] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4966.427903] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4971.502836] Lustre: Mounted lustre-client [ 4971.786121] LustreError: 305957:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff91afc8271800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 4971.811360] Lustre: Unmounted lustre-client [ 4976.609832] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4976.610221] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4976.612723] LustreError: Skipped 1 previous similar message [ 4976.613269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4976.618047] Lustre: Skipped 1 previous similar message [ 4980.685205] Lustre: server umount lustre-OST0000 complete [ 4986.335845] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4986.337949] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4986.342045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4986.344038] Lustre: Skipped 1 previous similar message [ 4987.991298] Lustre: server umount lustre-MDT0000 complete [ 4989.178212] LustreError: 303459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776874525 with bad export cookie 12200714591720266713 [ 4989.180577] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4989.181620] LustreError: 303459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4989.303395] Lustre: server umount lustre-MDT0001 complete [ 4992.744272] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 4993.751268] Key type lgssc unregistered [ 5021.535350] LNet: 307235:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5021.537181] LNetError: 307235:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5093.221349] LNet: Removed LNI 192.168.202.146@tcp [ 5093.517860] Key type .llcrypt unregistered [ 5093.519081] Key type ._llcrypt unregistered [ 5099.758398] Key type ._llcrypt registered [ 5099.759645] Key type .llcrypt registered [ 5099.799570] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 5106.297132] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 5106.591243] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5106.649100] alg: No test for adler32 (adler32-zlib) [ 5107.497709] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 5107.582657] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 5109.159166] Key type lgssc registered [ 5109.523284] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5119.152391] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 12:17:34 (1776874654) [ 5121.338351] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5121.343013] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5122.429561] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5122.440059] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5122.477551] Lustre: lustre-MDT0000: new disk, initializing [ 5122.500136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5122.506586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5123.759821] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 5127.777733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5127.814620] Lustre: 310398:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5127.833418] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5127.835117] Lustre: Skipped 1 previous similar message [ 5127.866269] Lustre: lustre-MDT0001: new disk, initializing [ 5127.893936] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5127.904114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5127.908715] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5129.126762] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 5132.154884] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5133.329569] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5135.485632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5135.575137] Lustre: lustre-OST0000: new disk, initializing [ 5135.577092] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5135.602596] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5137.340973] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 5137.782083] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5137.786321] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5137.814436] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5140.357283] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5148.129054] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5148.129870] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5148.131981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5148.134924] Lustre: Skipped 1 previous similar message [ 5150.286742] Lustre: server umount lustre-OST0000 complete [ 5153.248595] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5153.249350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5153.251881] Lustre: Skipped 1 previous similar message [ 5153.254735] Lustre: Skipped 2 previous similar messages [ 5157.653920] Lustre: server umount lustre-MDT0000 complete [ 5158.987231] LustreError: 312743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776874694 with bad export cookie 9166597839074223770 [ 5158.990682] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5158.991917] LustreError: 312743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5159.114227] Lustre: server umount lustre-MDT0001 complete [ 5162.794187] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 5163.805781] Key type lgssc unregistered [ 5163.937419] LNet: 313765:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5163.940235] LNetError: 313765:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5163.946326] LNet: Removed LNI 192.168.202.146@tcp [ 5164.249928] Key type .llcrypt unregistered [ 5164.251088] Key type ._llcrypt unregistered [ 5174.035804] Key type ._llcrypt registered [ 5174.036941] Key type .llcrypt registered [ 5174.082264] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 5174.505303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5174.534139] alg: No test for adler32 (adler32-zlib) [ 5175.391780] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 5175.470137] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 5177.047142] Key type lgssc registered [ 5177.432061] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5180.297626] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5180.303627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5181.497262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5182.897433] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 5185.879064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5186.006940] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5187.340435] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 5189.143915] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5190.338311] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5192.531895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5192.647580] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5194.543521] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 5196.428175] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5214.729434] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5215.277639] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5216.735923] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5216.738471] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5216.742138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5217.248297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5217.250177] Lustre: Skipped 1 previous similar message [ 5217.250199] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5217.255221] Lustre: Skipped 1 previous similar message [ 5218.709577] Lustre: server umount lustre-MDT0000 complete [ 5219.975225] LustreError: 315125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776874755 with bad export cookie 9529585292271632637 [ 5219.978828] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5219.978838] LustreError: 315125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5222.123241] Lustre: server umount lustre-MDT0001 complete [ 5239.767057] Lustre: server umount lustre-OST0000 complete [ 5242.777015] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5244.158353] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 5245.143354] Lustre: 318892:0:(mgs_llog.c:1798:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.146@tcp [ 5245.872605] Lustre: server umount MGS complete [ 5248.717851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5248.882989] LustreError: 319463:0:(ldlm_lib.c:1180:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5248.889047] LustreError: 319463:0:(ldlm_lib.c:1180:target_handle_connect()) Skipped 2 previous similar messages [ 5248.906125] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5250.217251] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 5253.118728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5253.232589] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5254.508149] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 5257.312188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5259.214613] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 5260.947079] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5261.510841] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [ 5262.723090] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5263.308758] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5274.085968] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5274.086696] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5274.089274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5274.092651] Lustre: Skipped 1 previous similar message [ 5274.097982] Lustre: Skipped 2 previous similar messages [ 5275.473374] Lustre: server umount lustre-OST0000 complete [ 5279.201189] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5279.201533] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5279.203617] LustreError: Skipped 2 previous similar messages [ 5279.210185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5279.213591] Lustre: Skipped 1 previous similar message [ 5282.900891] Lustre: server umount lustre-MDT0000 complete [ 5284.146124] LustreError: 319445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776874820 with bad export cookie 9529585292271633463 [ 5284.148040] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5284.151763] LustreError: 319445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5284.258538] Lustre: server umount lustre-MDT0001 complete [ 5288.086455] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 5289.183505] Key type lgssc unregistered [ 5289.317335] LNet: 322635:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5289.319808] LNetError: 322635:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5289.326285] LNet: Removed LNI 192.168.202.146@tcp [ 5289.628381] Key type .llcrypt unregistered [ 5289.629846] Key type ._llcrypt unregistered [ 5296.023623] Key type ._llcrypt registered [ 5296.024578] Key type .llcrypt registered [ 5296.069682] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 5302.645715] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 5302.976763] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5302.982454] alg: No test for adler32 (adler32-zlib) [ 5303.824945] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 5303.897518] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 5305.471213] Key type lgssc registered [ 5305.876859] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5317.325706] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5317.332583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5318.420851] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5318.431598] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5318.466193] Lustre: lustre-MDT0000: new disk, initializing [ 5318.490381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5318.497160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5319.868830] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 5324.064236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5324.106353] Lustre: 325734:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5324.131476] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5324.133046] Lustre: Skipped 1 previous similar message [ 5324.167257] Lustre: lustre-MDT0001: new disk, initializing [ 5324.198376] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5324.210468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5324.213711] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5325.495914] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 5328.615791] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5329.868335] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5332.123279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5332.223580] Lustre: lustre-OST0000: new disk, initializing [ 5332.225672] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5332.253854] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5334.211574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5334.217152] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5334.235039] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 5334.261569] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5337.480690] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5338.815524] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5338.886262] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5340.219633] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5340.294650] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5344.738633] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5344.738846] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5344.741468] LustreError: Skipped 1 previous similar message [ 5344.746242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5344.749359] Lustre: Skipped 1 previous similar message [ 5347.033920] Lustre: server umount lustre-OST0000 complete [ 5349.856654] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5349.857341] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5349.863438] Lustre: Skipped 1 previous similar message [ 5349.865970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5349.868130] Lustre: Skipped 1 previous similar message [ 5354.397270] Lustre: server umount lustre-MDT0000 complete [ 5355.623113] LustreError: 325725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776874891 with bad export cookie 206714503426673445 [ 5355.625906] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5355.626649] LustreError: 325725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5355.740214] Lustre: server umount lustre-MDT0001 complete [ 5358.676612] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 12:21:33 (1776874893) [ 5363.853288] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 5367.112200] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 5379.116243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5379.203122] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5379.213595] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5379.251330] Lustre: lustre-MDT0000: new disk, initializing [ 5379.277153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5379.283039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5380.609155] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 5384.824726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5384.866084] Lustre: 332041:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5384.889718] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5384.891423] Lustre: Skipped 1 previous similar message [ 5384.928561] Lustre: lustre-MDT0001: new disk, initializing [ 5384.959105] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5384.973754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5384.978621] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5386.378440] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 5389.596975] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5390.869789] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5393.143539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5393.259659] Lustre: lustre-OST0000: new disk, initializing [ 5393.261575] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5393.292965] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5393.490319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5393.493681] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5393.538067] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5395.312061] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 5398.447328] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5409.250301] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5409.250497] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5409.252992] LustreError: Skipped 1 previous similar message [ 5409.256207] Lustre: Skipped 1 previous similar message [ 5409.256834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5409.260372] Lustre: Skipped 1 previous similar message [ 5414.225949] Lustre: server umount lustre-OST0000 complete [ 5415.903873] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5415.907905] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5415.911216] Lustre: Skipped 1 previous similar message [ 5415.912924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5421.605445] Lustre: server umount lustre-MDT0000 complete [ 5422.845179] LustreError: 332034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776874958 with bad export cookie 206714503426674313 [ 5422.847489] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5422.848370] LustreError: 332034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5422.961574] Lustre: server umount lustre-MDT0001 complete [ 5426.839983] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 5427.946228] Key type lgssc unregistered [ 5428.080611] LNet: 335802:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5428.083445] LNetError: 335802:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5428.095457] LNet: Removed LNI 192.168.202.146@tcp [ 5428.384952] Key type .llcrypt unregistered [ 5428.386232] Key type ._llcrypt unregistered [ 5460.889688] Key type ._llcrypt registered [ 5460.890835] Key type .llcrypt registered [ 5460.927162] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 5461.023189] Key type .llcrypt unregistered [ 5461.024239] Key type ._llcrypt unregistered [ 5488.675447] Key type ._llcrypt registered [ 5488.676627] Key type .llcrypt registered [ 5488.721925] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 5488.845359] Key type .llcrypt unregistered [ 5488.846639] Key type ._llcrypt unregistered [ 5499.982739] Key type ._llcrypt registered [ 5499.983794] Key type .llcrypt registered [ 5500.023614] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 5500.128218] Key type .llcrypt unregistered [ 5500.129285] Key type ._llcrypt unregistered [ 5511.348509] Key type ._llcrypt registered [ 5511.349672] Key type .llcrypt registered [ 5511.387373] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 5511.482288] Key type .llcrypt unregistered [ 5511.483562] Key type ._llcrypt unregistered [ 5539.087646] Key type ._llcrypt registered [ 5539.088834] Key type .llcrypt registered [ 5539.125383] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 5539.220377] Key type .llcrypt unregistered [ 5539.221545] Key type ._llcrypt unregistered [ 5550.328888] Key type ._llcrypt registered [ 5550.329890] Key type .llcrypt registered [ 5550.371729] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 5550.470375] Key type .llcrypt unregistered [ 5550.471396] Key type ._llcrypt unregistered [ 5561.274894] Key type ._llcrypt registered [ 5561.275912] Key type .llcrypt registered [ 5561.309227] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 5561.399399] Key type .llcrypt unregistered [ 5561.401029] Key type ._llcrypt unregistered [ 5572.167811] Key type ._llcrypt registered [ 5572.168786] Key type .llcrypt registered [ 5572.205748] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 5572.312290] Key type .llcrypt unregistered [ 5572.313564] Key type ._llcrypt unregistered [ 5583.567200] Key type ._llcrypt registered [ 5583.568225] Key type .llcrypt registered [ 5583.604640] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 5583.705427] Key type .llcrypt unregistered [ 5583.706884] Key type ._llcrypt unregistered [ 5594.477207] Key type ._llcrypt registered [ 5594.478085] Key type .llcrypt registered [ 5594.516276] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 5594.623410] Key type .llcrypt unregistered [ 5594.625461] Key type ._llcrypt unregistered [ 5615.791038] Key type ._llcrypt registered [ 5615.792132] Key type .llcrypt registered [ 5615.824193] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 5615.913164] Key type .llcrypt unregistered [ 5615.914259] Key type ._llcrypt unregistered [ 5627.284587] Key type ._llcrypt registered [ 5627.285943] Key type .llcrypt registered [ 5627.330291] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 5627.439303] Key type .llcrypt unregistered [ 5627.440505] Key type ._llcrypt unregistered [ 5638.682761] Key type ._llcrypt registered [ 5638.684488] Key type .llcrypt registered [ 5638.729768] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 5638.835337] Key type .llcrypt unregistered [ 5638.836471] Key type ._llcrypt unregistered [ 5656.672449] Key type ._llcrypt registered [ 5656.673336] Key type .llcrypt registered [ 5656.712325] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 5663.147476] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 5663.703441] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5663.720968] alg: No test for adler32 (adler32-zlib) [ 5664.578341] Lustre: Lustre: Build Version: 2.17.52_53_gafd89ce [ 5664.660681] LNet: Added LNI 192.168.202.146@tcp [8/256/0/180] [ 5666.247141] Key type lgssc registered [ 5666.619083] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5678.641170] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5678.646287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5679.759971] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 5679.774374] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 5679.818365] Lustre: tst250fs-MDT0000: new disk, initializing [ 5679.845424] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5679.851288] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5681.174284] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 5685.312072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5685.350777] Lustre: 355448:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5685.376571] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 5685.378736] Lustre: Skipped 1 previous similar message [ 5685.415020] Lustre: tst250fs-MDT0001: new disk, initializing [ 5685.450326] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5685.462840] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5685.465956] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5686.778242] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 5689.075183] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5691.297346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5691.403397] Lustre: tst250fs-OST0000: new disk, initializing [ 5691.405830] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 5691.437526] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5693.435676] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 5697.803337] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5697.867261] Lustre: tst250fs-OST0001: new disk, initializing [ 5697.869945] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 5697.906681] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5699.528127] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5699.531143] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5699.591848] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5699.596352] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5699.995771] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_default_debug -1 all [ 5702.606638] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 5714.915587] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5714.916781] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 5714.920153] Lustre: Skipped 3 previous similar messages [ 5714.922458] Lustre: Skipped 2 previous similar messages [ 5720.033760] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 5720.036925] Lustre: Skipped 4 previous similar messages [ 5725.152832] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 5725.155673] Lustre: Skipped 3 previous similar messages [ 5727.199104] Lustre: tst250fs-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5727.273452] Lustre: server umount tst250fs-MDT0000 complete [ 5728.426970] LustreError: 356227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1776875264 with bad export cookie 15277713401851305736 [ 5728.429136] LustreError: MGC192.168.202.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5728.430323] LustreError: 356227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5728.538169] Lustre: server umount tst250fs-MDT0001 complete [ 5739.458877] Lustre: server umount tst250fs-OST0000 complete [ 5750.937098] Lustre: server umount tst250fs-OST0001 complete [ 5756.962875] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 5759.732987] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 5769.394352] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 12:28:24 (1776875304) [ 5769.976543] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 5774.885569] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing set_hostid [ 5778.200882] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing load_modules_local [ 5788.370871] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 5632 sec ========== 12:28:43 (1776875323) [ 5788.987791] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:28:44 (1776875324) === [ 5790.193534] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:28:45 (1776875325) === [ 5803.851287] Lustre: DEBUG MARKER: oleg246-server.virtnet: executing unload_modules_local [ 5804.900233] Key type lgssc unregistered [ 5805.031296] LNet: 366902:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5805.034470] LNetError: 366902:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5805.041338] LNet: Removed LNI 192.168.202.146@tcp [ 5805.374112] Key type .llcrypt unregistered [ 5805.375169] Key type ._llcrypt unregistered