[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 515877274 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001019] APIC: Switch to symmetric I/O mode setup [ 0.003315] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010146] LSM: Security Framework initializing [ 0.011052] Yama: becoming mindful. [ 0.012045] SELinux: Initializing. [ 0.013071] *** VALIDATE selinux *** [ 0.022098] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027071] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028169] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029136] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030119] *** VALIDATE tmpfs *** [ 0.032135] *** VALIDATE proc *** [ 0.033236] *** VALIDATE cgroup *** [ 0.034011] *** VALIDATE cgroup2 *** [ 0.035306] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037170] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039031] Spectre V2 : User space: Vulnerable [ 0.040011] Speculative Store Bypass: Vulnerable [ 0.043237] debug: unmapping init [mem 0xffffffffa7059000-0xffffffffa7060fff] [ 0.045205] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046708] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047022] ... version: 2 [ 0.048013] ... bit width: 48 [ 0.049008] ... generic registers: 4 [ 0.050010] ... value mask: 0000ffffffffffff [ 0.051010] ... max period: 00007fffffffffff [ 0.052012] ... fixed-purpose events: 3 [ 0.053010] ... event mask: 000000070000000f [ 0.054285] rcu: Hierarchical SRCU implementation. [ 0.056429] smp: Bringing up secondary CPUs ... [ 0.057617] x86: Booting SMP configuration: [ 0.058030] .... node #0, CPUs: #1 #2 #3 [ 0.061217] smp: Brought up 1 node, 4 CPUs [ 0.063015] smpboot: Max logical packages: 1 [ 0.064018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.123541] node 0 deferred pages initialised in 56ms [ 0.126098] devtmpfs: initialized [ 0.127193] x86/mm: Memory block size: 128MB [ 0.129929] gcov: version magic: 0x41383552 [ 0.135465] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.136197] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.137326] pinctrl core: initialized pinctrl subsystem [ 0.138194] [ 0.138735] ************************************************************* [ 0.139010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.140011] ** ** [ 0.141007] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.142010] ** ** [ 0.143008] ** This means that this kernel is built to expose internal ** [ 0.144010] ** IOMMU data structures, which may compromise security on ** [ 0.145007] ** your system. ** [ 0.146013] ** ** [ 0.147014] ** If you see this message and you are not debugging the ** [ 0.148017] ** kernel, report this immediately to your vendor! ** [ 0.149015] ** ** [ 0.150014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.151013] ************************************************************* [ 0.152773] NET: Registered protocol family 16 [ 0.154330] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.155079] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.157049] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.160009] cpuidle: using governor menu [ 0.161699] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.163521] PCI: Using configuration type 1 for base access [ 0.166147] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.176130] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.179099] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.182072] cryptd: max_cpu_qlen set to 1000 [ 0.184308] ACPI: Added _OSI(Module Device) [ 0.186019] ACPI: Added _OSI(Processor Device) [ 0.188012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.189014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.194048] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.200597] ACPI: Interpreter enabled [ 0.202069] ACPI: PM: (supports S0 S3 S4 S5) [ 0.203013] ACPI: Using IOAPIC for interrupt routing [ 0.205142] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.208431] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.218672] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.221055] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.223028] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.226164] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.231516] acpiphp: Slot [2] registered [ 0.233154] acpiphp: Slot [5] registered [ 0.234157] acpiphp: Slot [6] registered [ 0.236192] acpiphp: Slot [7] registered [ 0.237116] acpiphp: Slot [8] registered [ 0.238103] acpiphp: Slot [9] registered [ 0.239107] acpiphp: Slot [10] registered [ 0.240074] acpiphp: Slot [3] registered [ 0.241008] acpiphp: Slot [4] registered [ 0.242035] acpiphp: Slot [11] registered [ 0.243203] acpiphp: Slot [12] registered [ 0.245107] acpiphp: Slot [13] registered [ 0.246146] acpiphp: Slot [14] registered [ 0.248115] acpiphp: Slot [15] registered [ 0.249165] acpiphp: Slot [16] registered [ 0.251144] acpiphp: Slot [17] registered [ 0.252132] acpiphp: Slot [18] registered [ 0.254140] acpiphp: Slot [19] registered [ 0.255219] acpiphp: Slot [20] registered [ 0.257133] acpiphp: Slot [21] registered [ 0.258108] acpiphp: Slot [22] registered [ 0.260104] acpiphp: Slot [23] registered [ 0.261130] acpiphp: Slot [24] registered [ 0.263136] acpiphp: Slot [25] registered [ 0.264114] acpiphp: Slot [26] registered [ 0.266104] acpiphp: Slot [27] registered [ 0.267140] acpiphp: Slot [28] registered [ 0.269097] acpiphp: Slot [29] registered [ 0.270107] acpiphp: Slot [30] registered [ 0.272127] acpiphp: Slot [31] registered [ 0.273070] PCI host bridge to bus 0000:00 [ 0.275023] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.277026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.279022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.282026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.284025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.287063] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.289207] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.292549] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.295391] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.304495] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.310065] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.313021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.315019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.317017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.321564] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.323077] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.326045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.328918] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.334017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.348019] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.353023] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.357940] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.362015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.367016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.381022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.391271] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.403019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.415026] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.434027] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.445036] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.452019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.456018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.472034] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.482135] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.488024] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.495016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.511033] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.519883] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.526018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.532024] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.546021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.555564] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.562016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.570018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.602023] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.618063] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.620283] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.622310] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.623302] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.625233] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.628188] iommu: Default domain type: Passthrough [ 0.629431] SCSI subsystem initialized [ 0.631116] ACPI: bus type USB registered [ 0.632086] usbcore: registered new interface driver usbfs [ 0.633048] usbcore: registered new interface driver hub [ 0.634155] usbcore: registered new device driver usb [ 0.635156] pps_core: LinuxPPS API ver. 1 registered [ 0.637007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.639102] PTP clock support registered [ 0.640179] EDAC MC: Ver: 3.0.0 [ 0.642106] PCI: Using ACPI for IRQ routing [ 0.643837] NetLabel: Initializing [ 0.645009] NetLabel: domain hash size = 128 [ 0.645944] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.647330] NetLabel: unlabeled traffic allowed by default [ 0.649111] vgaarb: loaded [ 0.650237] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.651008] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.655370] clocksource: Switched to clocksource kvm-clock [ 0.759923] VFS: Disk quotas dquot_6.6.0 [ 0.761552] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.764112] *** VALIDATE ramfs *** [ 0.765442] *** VALIDATE hugetlbfs *** [ 0.767207] pnp: PnP ACPI init [ 0.769624] pnp: PnP ACPI: found 6 devices [ 0.792929] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.800354] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.804078] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.807039] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.809108] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.811228] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.813662] NET: Registered protocol family 2 [ 0.815808] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.820051] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.823143] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.827692] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.831841] TCP: Hash tables configured (established 65536 bind 65536) [ 0.834912] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.837950] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.840477] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.843037] NET: Registered protocol family 1 [ 0.845586] RPC: Registered named UNIX socket transport module. [ 0.847079] RPC: Registered udp transport module. [ 0.848859] RPC: Registered tcp transport module. [ 0.850735] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.852628] NET: Registered protocol family 44 [ 0.854361] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.856471] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.858649] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.861153] PCI: CLS 0 bytes, default 64 [ 0.863727] Unpacking initramfs... [ 3.168816] debug: unmapping init [mem 0xffff9393bcc54000-0xffff9393bffbffff] [ 3.177686] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.182840] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.194059] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 5.398684] Initialise system trusted keyrings [ 5.402478] Key type blacklist registered [ 5.421829] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 5.445714] zbud: loaded [ 5.461543] *** VALIDATE nfs *** [ 5.464441] *** VALIDATE nfs4 *** [ 5.467407] pstore: using deflate compression [ 5.471602] Platform Keyring initialized [ 5.776976] NET: Registered protocol family 38 [ 5.784016] Key type asymmetric registered [ 5.789771] Asymmetric key parser 'x509' registered [ 5.793705] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 5.806624] io scheduler mq-deadline registered [ 5.808794] io scheduler kyber registered [ 5.811508] io scheduler bfq registered [ 5.814150] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 5.818940] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 5.822890] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 5.827576] ACPI: Power Button [PWRF] [ 5.840227] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 5.854198] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 5.886362] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 5.910958] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 5.941332] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.973568] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 6.006537] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 6.017566] Non-volatile memory driver v1.3 [ 6.021992] Linux agpgart interface v0.103 [ 6.081580] virtio_blk virtio1: [vda] 146016 512-byte logical blocks (74.8 MB/71.3 MiB) [ 6.086308] vda: detected capacity change from 0 to 74760192 [ 6.141522] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 6.146781] vdb: detected capacity change from 0 to 1073741824 [ 6.185232] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.190643] vdc: detected capacity change from 0 to 2621440000 [ 6.216840] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 6.222380] vdd: detected capacity change from 0 to 2621440000 [ 6.253317] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.258588] vde: detected capacity change from 0 to 4294967296 [ 6.290746] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 6.302315] vdf: detected capacity change from 0 to 4294967296 [ 6.319748] libphy: Fixed MDIO Bus: probed [ 6.333612] usbcore: registered new interface driver usbserial_generic [ 6.336390] usbserial: USB Serial support registered for generic [ 6.339727] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 6.345866] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 6.348760] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 6.352332] mousedev: PS/2 mouse device common for all mice [ 6.357846] rtc_cmos 00:05: RTC can wake from S4 [ 6.360941] rtc_cmos 00:05: registered as rtc0 [ 6.363779] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 6.368942] intel_pstate: CPU model not supported [ 6.373919] hid: raw HID events driver (C) Jiri Kosina [ 6.375295] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 6.377747] usbcore: registered new interface driver usbhid [ 6.377757] usbhid: USB HID core driver [ 6.377926] drop_monitor: Initializing network drop monitor service [ 6.378492] Initializing XFRM netlink socket [ 6.378884] NET: Registered protocol family 10 [ 6.381914] Segment Routing with IPv6 [ 6.395741] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 6.401587] NET: Registered protocol family 17 [ 6.410194] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 6.415587] mpls_gso: MPLS GSO support [ 6.436098] RAS: Correctable Errors collector initialized. [ 6.438955] AVX version of gcm_enc/dec engaged. [ 6.440857] AES CTR mode by8 optimization enabled [ 6.618823] sched_clock: Marking stable (6618788405, 0)->(7511580334, -892791929) [ 6.627703] registered taskstats version 1 [ 6.630728] Loading compiled-in X.509 certificates [ 6.633197] zswap: loaded using pool lzo/zbud [ 6.689711] Key type big_key registered [ 6.714187] Key type encrypted registered [ 6.716677] ima: No TPM chip found, activating TPM-bypass! [ 6.719258] ima: Allocated hash algorithm: sha1 [ 6.721682] ima: No architecture policies found [ 6.723239] evm: Initialising EVM extended attributes: [ 6.725270] evm: security.selinux [ 6.726797] evm: security.ima [ 6.728298] evm: security.capability [ 6.730737] evm: HMAC attrs: 0x1 [ 6.733964] rtc_cmos 00:05: setting system clock to 2026-08-29 08:42:37 UTC (1787992957) [ 6.742787] debug: unmapping init [mem 0xffffffffa8003000-0xffffffffa81fffff] [ 6.746819] debug: unmapping init [mem 0xffffffffa6d82000-0xffffffffa7058fff] [ 6.763758] Write protecting the kernel read-only data: 28672k [ 6.779539] debug: unmapping init [mem 0xffffffffa5403000-0xffffffffa55fffff] [ 6.797547] debug: unmapping init [mem 0xffffffffa5d14000-0xffffffffa5dfffff] [ 6.947878] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 6.963995] systemd[1]: Detected virtualization kvm. [ 6.966880] systemd[1]: Detected architecture x86-64. [ 6.970285] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 7.041962] systemd[1]: No hostname configured. [ 7.043958] systemd[1]: Set hostname to . [ 7.049303] random: systemd: uninitialized urandom read (16 bytes read) [ 7.061990] systemd[1]: Initializing machine ID from random generator. [ 7.235312] random: ln: uninitialized urandom read (6 bytes read) [ 7.476847] random: systemd: uninitialized urandom read (16 bytes read) [ 7.482292] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 7.498360] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 7.508809] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Swap. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 8.912867] device-mapper: uevent: version 1.0.3 [ 8.924252] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 10.544693] virtio_net virtio0 ens2: renamed from eth0 Starting dracut initqueue hook... [ 10.608003] random: fast init done [ 10.792647] scsi host0: ata_piix [ 10.829543] scsi host1: ata_piix [ 10.843400] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 10.851652] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 16.249977] random: crng init done [ 16.251647] random: 7 urandom warning(s) missed due to ratelimiting [ 18.253698] dracut-initqueue[591]: RTNETLINK answers: File exists [ 19.015118] hrtimer: interrupt took 3092427 ns Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 19.809624] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 22.408499] printk: systemd: 26 output lines suppressed due to ratelimiting [ 23.079741] SELinux: Disabled at runtime. [ 23.214345] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 23.226825] systemd[1]: Detected virtualization kvm. [ 23.229756] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 25.746844] systemd[1]: initrd-switch-root.service: Succeeded. [ 25.754867] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 25.769183] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 25.776855] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 25.794830] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 25.814560] systemd[1]: Starting Journal Service... Starting Journal Service... [ 25.835640] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Apply Kernel Variables... Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Paths. Mounting Kernel Debug File System... [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ 26.527310] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Control Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 27.646593] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 28.879281] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 28.922937] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 29.535477] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 29.835227] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit)[ 33.701413] Key type dns_resolver registered [** ] A start job is running for Configur…-only root support (8s / no limit)[ 34.271685] NFS: Registering the id_resolver key type [ 34.275130] Key type id_resolver registered [ 34.277723] Key type id_legacy registered [*** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Load/Save Random Seed. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started irqbalance daemon. [ OK ] Started dnf makecache --timer. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg228-server login: [ 98.137459] libcfs: loading out-of-tree module taints kernel. [ 98.170063] Key type ._llcrypt registered [ 98.171727] Key type .llcrypt registered [ 98.285948] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 118.809345] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 121.109787] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 121.143279] alg: No test for adler32 (adler32-zlib) [ 122.478666] Lustre: Lustre: Build Version: 2.17.57_44_gac22094 [ 123.695909] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 125.663316] Key type lgssc registered [ 127.166383] Lustre: Echo OBD driver; http://www.lustre.org/ [ 146.935562] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 191.127514] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 204.849624] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 204.925921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 206.281888] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 206.320607] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 206.473381] Lustre: lustre-MDT0000: new disk, initializing [ 206.617902] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 206.655624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 211.145302] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 226.726846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 226.880180] Lustre: 6513:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 226.925354] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 226.930827] Lustre: Skipped 1 previous similar message [ 227.059603] Lustre: lustre-MDT0001: new disk, initializing [ 227.120078] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 227.153986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 227.163887] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 232.306097] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 237.947521] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 249.004881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 249.409656] Lustre: lustre-OST0000: new disk, initializing [ 249.415274] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 249.424438] Lustre: 8449:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 249.528674] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 256.344271] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 257.567959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 257.591115] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 257.668847] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 271.932074] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 272.087389] Lustre: lustre-OST0001: new disk, initializing [ 272.092726] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 272.098724] Lustre: 9523:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 272.155796] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 277.976366] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 278.059884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 278.069934] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 278.190106] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 289.765736] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 295.130681] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 302.101213] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing check_logdir /tmp/testlogs/ [ 307.296951] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing yml_node [ 310.979669] Lustre: DEBUG MARKER: Client: 2.17.57.44 [ 313.538123] Lustre: DEBUG MARKER: MDS: 2.17.57.44 [ 315.910899] Lustre: DEBUG MARKER: OSS: 2.17.57.44 [ 317.412513] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 29 04:47:46 EDT 2026 [ 335.599585] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 337.117649] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 354.787884] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 354.796196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 354.806730] Lustre: Skipped 2 previous similar messages [ 354.827880] Lustre: Skipped 3 previous similar messages [ 359.418389] Lustre: server umount lustre-MDT0000 complete [ 362.817774] LustreError: 6504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787993313 with bad export cookie 983351775961139412 [ 362.822863] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 362.829627] LustreError: 6504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 365.024354] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 365.026856] LustreError: 6520:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 365.069768] Lustre: Skipped 1 previous similar message [ 365.072623] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 365.099086] LustreError: 6520:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 365.118518] Lustre: Skipped 1 previous similar message [ 367.073780] LustreError: 6520:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 367.073808] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 367.103451] LustreError: 6520:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 367.120424] Lustre: Skipped 1 previous similar message [ 372.199036] LustreError: 6519:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 372.209204] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 372.247396] Lustre: Skipped 1 previous similar message [ 377.311134] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 377.635140] Lustre: server umount lustre-MDT0001 complete [ 381.547511] Lustre: server umount lustre-OST0000 complete [ 385.951630] Lustre: server umount lustre-OST0001 complete [ 392.936034] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 401.924593] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 438.201504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 438.452665] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 438.499678] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 438.622354] Lustre: lustre-MDT0000: new disk, initializing [ 438.720288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 438.739465] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 444.024296] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 454.697842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 454.812610] Lustre: 15076:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 454.864745] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 454.868193] Lustre: Skipped 1 previous similar message [ 454.923219] Lustre: lustre-MDT0001: new disk, initializing [ 454.988540] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 455.035489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 455.063223] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 459.328467] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 469.487735] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 477.438545] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 485.648322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 485.905689] Lustre: lustre-OST0000: new disk, initializing [ 485.908734] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 485.914637] Lustre: 17049:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 486.024570] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 487.839476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 487.858407] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 488.052511] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 493.190488] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 503.647349] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 508.599195] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 508.836892] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 513.789067] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 514.089818] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 518.628905] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 518.633557] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 518.634356] LustreError: Skipped 1 previous similar message [ 518.635082] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 518.635086] Lustre: Skipped 2 previous similar messages [ 518.674209] Lustre: Skipped 2 previous similar messages [ 522.728910] Lustre: server umount lustre-OST0000 complete [ 526.817088] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 526.822876] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 526.832974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 526.835689] Lustre: Skipped 1 previous similar message [ 541.151395] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 541.750325] Lustre: server umount lustre-MDT0000 complete [ 544.225569] LustreError: 15958:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 544.245394] LustreError: 15958:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 545.999495] LustreError: 15071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787993496 with bad export cookie 983351775961140952 [ 546.006135] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 546.008344] LustreError: 15071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 546.643870] Lustre: server umount lustre-MDT0001 complete [ 552.535830] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 04:51:41 (1787993501) [ 576.142868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 576.806887] LustreError: 19926:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 576.889778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 583.910054] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 587.238505] LustreError: 19926:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 587.246314] LustreError: 19926:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 593.906929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 598.581061] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 607.327808] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 613.027645] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 615.903194] Lustre: Failing over lustre-MDT0000 [ 615.916702] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 615.935829] Lustre: Skipped 1 previous similar message [ 615.953268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 615.965306] Lustre: Skipped 7 previous similar messages [ 616.183472] Lustre: server umount lustre-MDT0000 complete [ 619.041433] LustreError: 21129:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 619.069726] LustreError: 21129:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 634.337447] Lustre: 3647:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787993569/real 1787993569] req@ffff939307446d80 x1874846625725184/t0(0) o400->MGC192.168.202.128@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787993585 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 634.359297] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 636.756770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 645.600311] LustreError: 3644:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff939306d71c00 x1874846625727872/t0(0) o250->MGC192.168.202.128@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 645.669913] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 645.691700] Lustre: MGS: Client aec4dc7f-816b-4c18-9520-ed3b69d0400c (at 0@lo) reconnecting [ 645.706183] Lustre: MGC192.168.202.128@tcp: Connection restored to 0@lo (at 0@lo) [ 646.014274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 646.023188] Lustre: Skipped 1 previous similar message [ 646.094373] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 651.238019] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 651.238776] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 651.251098] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 651.274535] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 662.033474] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 666.596118] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 666.599550] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 666.630481] Lustre: Skipped 1 previous similar message [ 666.640538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 672.521659] Lustre: server umount lustre-MDT0000 complete [ 673.763101] LustreError: 19927:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 673.796569] LustreError: 19927:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 676.035666] LustreError: 19907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787993626 with bad export cookie 983351775961142366 [ 676.054224] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 676.284715] Lustre: server umount lustre-MDT0001 complete [ 685.841246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 691.289386] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 702.189964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 707.108571] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 715.153977] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 722.381899] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 725.307458] Lustre: Failing over lustre-MDT0000 [ 725.932625] Lustre: server umount lustre-MDT0000 complete [ 727.403509] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 727.426693] Lustre: Skipped 1 previous similar message [ 737.762436] LustreError: 23265:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 737.775600] LustreError: 23265:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 743.839137] Lustre: 3646:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787993678/real 1787993678] req@ffff9393073ec380 x1874846625756416/t0(0) o400->MGC192.168.202.128@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787993694 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 743.880235] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 749.377910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 754.144143] LustreError: 3644:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff939306d70380 x1874846625759360/t0(0) o250->MGC192.168.202.128@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 754.783077] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 759.789019] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 759.807547] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 759.818058] Lustre: Skipped 1 previous similar message [ 759.846134] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 761.522597] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 773.773636] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 778.216571] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 778.225166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 778.238447] Lustre: Skipped 2 previous similar messages [ 778.257741] Lustre: Skipped 3 previous similar messages [ 784.384219] Lustre: server umount lustre-MDT0000 complete [ 788.738075] LustreError: 23245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787993739 with bad export cookie 983351775961143311 [ 788.742036] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 788.754796] LustreError: 23245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 789.257195] Lustre: server umount lustre-MDT0001 complete [ 797.822627] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 04:55:46 (1787993746) [ 814.934707] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 826.425112] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 869.525853] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 880.682971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 880.869818] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 880.890121] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 881.050408] Lustre: lustre-MDT0000: new disk, initializing [ 881.204180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 881.211897] Lustre: Skipped 3 previous similar messages [ 881.267320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 885.520151] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 896.499960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 896.634698] Lustre: 29639:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 896.881676] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 900.983260] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 910.173490] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 916.416604] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 922.983310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 923.237833] Lustre: 31609:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 924.431934] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 924.470044] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 924.588200] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 924.618949] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 929.742333] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 939.872529] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 946.596293] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 965.621599] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 965.638145] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 965.669661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 965.679842] Lustre: Skipped 2 previous similar messages [ 969.703776] Lustre: server umount lustre-OST0000 complete [ 980.947933] Lustre: server umount lustre-MDT0000 complete [ 980.970472] LustreError: 32330:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 980.996543] LustreError: 32330:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 985.653914] LustreError: 31608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787993936 with bad export cookie 983351775961143717 [ 985.662788] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 986.391466] Lustre: server umount lustre-MDT0001 complete [ 1001.384846] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 1004.762858] Key type lgssc unregistered [ 1005.211292] LNet: 33594:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1005.217896] LNetError: 33594:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1005.235748] LNet: Removed LNI 192.168.202.128@tcp [ 1006.405266] Key type .llcrypt unregistered [ 1006.407152] Key type ._llcrypt unregistered [ 1032.193360] Key type ._llcrypt registered [ 1032.195362] Key type .llcrypt registered [ 1032.273514] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 1033.266495] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1033.371804] alg: No test for adler32 (adler32-zlib) [ 1034.536351] Lustre: Lustre: Build Version: 2.17.57_44_gac22094 [ 1034.881908] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 1036.583189] Key type lgssc registered [ 1037.676892] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1045.583295] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1045.604975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1047.175108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1052.503541] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1056.751392] LustreError: 35010:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1056.772658] LustreError: 35010:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1061.863372] LustreError: 35011:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1063.218714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1063.685135] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1068.764323] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1076.301080] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1082.290447] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1089.409784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1089.979046] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1092.101207] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1096.966835] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1105.712884] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1108.402452] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1122.017980] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 05:01:10 (1787994070) [ 1125.048423] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1127.117846] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1134.433634] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 05:01:22 (1787994082) [ 1144.201984] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 05:01:32 (1787994092) [ 1145.924327] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1157.138732] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1157.146914] Lustre: Skipped 2 previous similar messages [ 1159.207230] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 05:01:47 (1787994107) [ 1161.214813] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1161.219150] Lustre: Skipped 2 previous similar messages [ 1170.408582] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1170.412616] Lustre: Skipped 4 previous similar messages [ 1187.397954] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 05:02:15 (1787994135) [ 1190.847062] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1195.179630] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1205.118149] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 05:02:33 (1787994153) [ 1208.252839] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1208.257062] Lustre: Skipped 6 previous similar messages [ 1222.807977] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 05:02:51 (1787994171) [ 1240.547861] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1240.561619] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1240.581882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1241.568566] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1241.579448] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1241.580328] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1241.599197] Lustre: Skipped 1 previous similar message [ 1243.861459] Lustre: server umount lustre-OST0000 complete [ 1251.812343] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1251.841522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1251.855642] Lustre: Skipped 1 previous similar message [ 1254.729532] Lustre: server umount lustre-MDT0000 complete [ 1256.930115] LustreError: 35011:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1256.978039] LustreError: 35011:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1258.909968] LustreError: 40054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787994209 with bad export cookie 8634384236291096988 [ 1258.920844] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1259.303689] Lustre: server umount lustre-MDT0001 complete [ 1270.528865] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 1274.135640] Key type lgssc unregistered [ 1274.461770] LNet: 42885:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1274.473982] LNetError: 42885:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1274.503397] LNet: Removed LNI 192.168.202.128@tcp [ 1275.667933] Key type .llcrypt unregistered [ 1275.669498] Key type ._llcrypt unregistered [ 1297.975687] Key type ._llcrypt registered [ 1297.984297] Key type .llcrypt registered [ 1298.110459] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 1314.655795] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 1315.967374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1316.224114] alg: No test for adler32 (adler32-zlib) [ 1317.391680] Lustre: Lustre: Build Version: 2.17.57_44_gac22094 [ 1317.601656] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 1319.256113] Key type lgssc registered [ 1320.490899] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1357.450655] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1357.495140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1358.718560] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1358.733807] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1358.800912] Lustre: lustre-MDT0000: new disk, initializing [ 1358.856558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1358.868042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1362.965700] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1374.590053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1374.817744] Lustre: 46039:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1374.924896] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1374.933671] Lustre: Skipped 1 previous similar message [ 1375.124191] Lustre: lustre-MDT0001: new disk, initializing [ 1375.290844] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1375.365972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1375.378170] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1380.013191] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1390.547153] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1397.027377] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1404.786062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1405.004828] Lustre: lustre-OST0000: new disk, initializing [ 1405.007173] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1405.011492] Lustre: 48010:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1405.088572] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1410.582981] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1414.737436] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1414.752856] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1414.934607] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1420.436618] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1424.860205] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1425.065926] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1429.987383] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1430.171834] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1435.616089] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1435.619500] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1435.627610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1435.665473] Lustre: Skipped 1 previous similar message [ 1440.744723] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1440.758365] Lustre: Skipped 2 previous similar messages [ 1445.857528] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1445.866939] Lustre: Skipped 1 previous similar message [ 1446.367109] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1446.582992] Lustre: server umount lustre-OST0000 complete [ 1452.000422] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1452.004563] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1452.025655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1456.097113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1456.099222] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1457.678929] Lustre: server umount lustre-MDT0000 complete [ 1461.800389] LustreError: 46032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787994412 with bad export cookie 11600125450453812291 [ 1461.813523] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1461.822155] LustreError: 46032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1462.200328] Lustre: server umount lustre-MDT0001 complete [ 1471.585809] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 05:07:00 (1787994420) [ 1478.683994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1478.938118] LustreError: 50479:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1478.953465] LustreError: 50479:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1479.008116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1483.380844] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1484.258682] LustreError: 50480:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1488.355753] LustreError: 50479:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1493.089890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1493.543678] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1498.892408] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1508.470085] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1514.488067] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1521.548874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1521.886698] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1530.396540] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1542.166810] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1549.123794] LustreError: 53008:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1550.237504] Lustre: Modifying parameter general.timeout=1 in log params [ 1552.349224] Lustre: Modifying parameter general.timeout=3 in log params [ 1552.352751] Lustre: Skipped 1 previous similar message [ 1554.402375] Lustre: Modifying parameter general.timeout=5 in log params [ 1554.411796] Lustre: Skipped 1 previous similar message [ 1559.102376] Lustre: Modifying parameter general.timeout=10 in log params [ 1559.117985] Lustre: Skipped 4 previous similar messages [ 1567.470368] Lustre: Modifying parameter general.timeout=19 in log params [ 1567.478171] Lustre: Skipped 8 previous similar messages [ 1584.065969] Lustre: Modifying parameter general.timeout=36 in log params [ 1584.070689] Lustre: Skipped 16 previous similar messages [ 1616.251748] Lustre: Modifying parameter general.timeout=71 in log params [ 1616.254168] Lustre: Skipped 34 previous similar messages [ 1679.840366] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 05:10:29 (1787994629) [ 1680.735389] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_policies=tbf in log params [ 1680.740315] Lustre: Skipped 59 previous similar messages [ 1682.747531] Lustre: 59963:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1684.278035] Lustre: 60062:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1687.661396] Lustre: 60259:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1687.673526] Lustre: 60259:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1692.067801] Lustre: 60471:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1692.082670] Lustre: 60471:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1723.649825] Lustre: 61835:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1723.665455] LustreError: 61835:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1727.917783] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 05:11:16 (1787994676) [ 1741.794804] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1741.805030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1741.816141] Lustre: Skipped 2 previous similar messages [ 1741.822781] Lustre: Skipped 3 previous similar messages [ 1747.078496] Lustre: server umount lustre-MDT0000 complete [ 1751.956782] LustreError: 50462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787994702 with bad export cookie 11600125450453813173 [ 1751.967034] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1752.034086] LustreError: 51769:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1752.041348] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1752.344684] Lustre: server umount lustre-MDT0001 complete [ 1758.777995] Lustre: server umount lustre-OST0000 complete [ 1785.704356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1786.049209] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1786.344572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1790.369785] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1799.805224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1799.856942] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1799.878129] Lustre: 64364:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1805.776817] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1814.546860] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1821.945393] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1830.122246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1830.299362] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1830.522744] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1830.528397] Lustre: Skipped 1 previous similar message [ 1837.673298] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1846.432284] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1848.683793] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1848.689804] Lustre: Skipped 22 previous similar messages [ 1848.746845] Lustre: 66929:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1856.480664] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1856.491233] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1856.497789] LustreError: Skipped 1 previous similar message [ 1856.498354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1856.498358] Lustre: Skipped 4 previous similar messages [ 1856.533302] Lustre: Skipped 1 previous similar message [ 1860.938560] Lustre: server umount lustre-OST0000 complete [ 1866.728297] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1871.848812] LustreError: 64373:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1871.983784] Lustre: server umount lustre-MDT0000 complete [ 1876.074299] LustreError: 64358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787994826 with bad export cookie 11600125450453816246 [ 1876.080984] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1876.087200] LustreError: 64358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1876.417523] Lustre: server umount lustre-MDT0001 complete [ 1887.079707] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 1890.078935] Key type lgssc unregistered [ 1890.341919] LNet: 68145:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1890.351422] LNetError: 68145:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1890.388915] LNet: Removed LNI 192.168.202.128@tcp [ 1891.783509] Key type .llcrypt unregistered [ 1891.787176] Key type ._llcrypt unregistered [ 1916.809510] Key type ._llcrypt registered [ 1916.815307] Key type .llcrypt registered [ 1916.927664] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 1918.434693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1918.454204] alg: No test for adler32 (adler32-zlib) [ 1919.663631] Lustre: Lustre: Build Version: 2.17.57_44_gac22094 [ 1919.944766] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 1921.783175] Key type lgssc registered [ 1923.019350] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1932.352465] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1932.371189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1933.937503] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1938.342695] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1943.529974] LustreError: 69559:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1943.546896] LustreError: 69559:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1946.455264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1947.037438] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1952.368568] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1959.867332] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1965.436800] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1972.986217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1973.606507] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1980.048215] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1988.349589] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2000.350704] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2012.642622] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2012.654914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2012.662311] Lustre: Skipped 2 previous similar messages [ 2012.677885] Lustre: Skipped 2 previous similar messages [ 2015.908337] Lustre: server umount lustre-MDT0000 complete [ 2017.763634] LustreError: 69555:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2017.788461] LustreError: 69555:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2019.265265] LustreError: 69540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787994970 with bad export cookie 15154596643170291132 [ 2019.285202] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2019.290302] LustreError: 69540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2019.678660] Lustre: server umount lustre-MDT0001 complete [ 2034.144522] Lustre: server umount lustre-OST0000 complete [ 2064.535817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2065.049579] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2065.129239] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2065.538145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2070.083554] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2078.933657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2079.107079] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2079.158512] Lustre: 74430:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2079.860206] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2085.107370] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2094.703631] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2102.529511] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2110.480284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2110.790731] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2111.363885] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2118.253551] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2125.566432] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2127.129080] Lustre: 77007:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2129.538022] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2141.157895] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2141.171850] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2141.195228] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2141.670696] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2143.577751] Lustre: server umount lustre-OST0000 complete [ 2151.393181] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2151.400848] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2151.413882] Lustre: Skipped 1 previous similar message [ 2151.424909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2151.430482] Lustre: Skipped 1 previous similar message [ 2154.197901] Lustre: server umount lustre-MDT0000 complete [ 2157.024925] LustreError: 74438:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2157.052710] LustreError: 74438:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2157.966973] LustreError: 74422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787995108 with bad export cookie 15154596643170291965 [ 2157.995604] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2158.002933] LustreError: 74422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2158.630953] Lustre: server umount lustre-MDT0001 complete [ 2170.837046] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 2173.869736] Key type lgssc unregistered [ 2174.290357] LNet: 78224:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2174.301829] LNetError: 78224:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2174.346181] LNet: Removed LNI 192.168.202.128@tcp [ 2175.247804] Key type .llcrypt unregistered [ 2175.251418] Key type ._llcrypt unregistered [ 2202.791733] Key type ._llcrypt registered [ 2202.793787] Key type .llcrypt registered [ 2202.967829] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 2204.084985] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2204.207684] alg: No test for adler32 (adler32-zlib) [ 2205.251528] Lustre: Lustre: Build Version: 2.17.57_44_gac22094 [ 2205.477534] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 2207.159312] Key type lgssc registered [ 2208.699437] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2216.709154] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2216.745089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2218.237254] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2222.926330] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2227.684968] LustreError: 79650:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2227.715422] LustreError: 79650:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2231.222452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2231.647718] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2236.124833] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2244.356570] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2251.637253] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2259.388976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2260.445044] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2266.634094] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2275.784618] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2282.647590] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2283.635120] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2285.319689] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2285.324342] Lustre: Skipped 1 previous similar message [ 2287.709428] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2287.714476] Lustre: Skipped 2 previous similar messages [ 2291.827643] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2291.832450] Lustre: Skipped 3 previous similar messages [ 2300.667827] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2300.672733] Lustre: Skipped 7 previous similar messages [ 2317.696836] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2317.700859] Lustre: Skipped 17 previous similar messages [ 2350.727917] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2350.735773] Lustre: Skipped 35 previous similar messages [ 2386.193269] Lustre: 87754:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2389.376253] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 05:22:17 (1787995337) [ 2391.355490] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2393.096953] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 05:22:21 (1787995341) [ 2399.717851] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2399.731798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2399.734732] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2403.871777] Lustre: server umount lustre-OST0000 complete [ 2409.955566] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2409.964741] Lustre: Skipped 1 previous similar message [ 2409.982458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2409.988191] Lustre: Skipped 2 previous similar messages [ 2414.032371] Lustre: server umount lustre-MDT0000 complete [ 2415.078996] LustreError: 80669:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2415.094466] LustreError: 80669:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2417.792610] LustreError: 82237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787995368 with bad export cookie 7891526886510469897 [ 2417.801539] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2417.803234] LustreError: 82237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2418.058454] Lustre: server umount lustre-MDT0001 complete [ 2423.519034] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 2424.560887] Key type lgssc unregistered [ 2424.694493] LNet: 89004:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2424.698406] LNetError: 89004:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2424.708514] LNet: Removed LNI 192.168.202.128@tcp [ 2425.106692] Key type .llcrypt unregistered [ 2425.108080] Key type ._llcrypt unregistered [ 2426.740209] Key type ._llcrypt registered [ 2426.741237] Key type .llcrypt registered [ 2426.793344] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2428.321594] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules [ 2428.641655] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2428.648865] alg: No test for adler32 (adler32-zlib) [ 2429.497378] Lustre: Lustre: Build Version: 2.17.57_44_gac22094 [ 2429.590204] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 2431.167161] Key type lgssc registered [ 2431.571830] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2431.955562] LustreError: 90157:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2439.490265] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 2442.839023] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2453.266877] LustreError: 90157:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2453.281050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2454.457049] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2455.836363] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2458.672549] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 05:23:28 (1787995408) [ 2460.139325] LustreError: 91212:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2460.142624] LustreError: 91212:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2460.145096] LustreError: 91212:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 5, retries 0, failed: rc = -5 [ 2460.261359] Lustre: server umount lustre-MDT0000 complete [ 2464.548858] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 2465.583572] Key type lgssc unregistered [ 2465.709326] LNet: 92706:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2465.712250] LNetError: 92706:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2465.719350] LNet: Removed LNI 192.168.202.128@tcp [ 2466.040711] Key type .llcrypt unregistered [ 2466.041830] Key type ._llcrypt unregistered [ 2472.867216] Key type ._llcrypt registered [ 2472.868214] Key type .llcrypt registered [ 2472.909442] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 2473.304165] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2473.371739] alg: No test for adler32 (adler32-zlib) [ 2474.242154] Lustre: Lustre: Build Version: 2.17.57_44_gac22094 [ 2474.335675] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 2475.919094] Key type lgssc registered [ 2476.263792] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2478.897197] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2478.903322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2480.082687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2481.502515] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2484.361965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2484.585299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2486.123325] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2488.741083] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2490.631599] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2493.134036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2493.379174] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2495.463391] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2498.091704] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2529.958600] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 05:24:39 (1787995479) [ 2530.533985] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2531.140955] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 05:24:40 (1787995480) [ 2535.395112] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2535.396560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2535.398681] Lustre: Skipped 1 previous similar message [ 2538.325395] Lustre: server umount lustre-MDT0000 complete [ 2539.655546] LustreError: 93881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787995490 with bad export cookie 1431848556132153340 [ 2539.658390] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2539.660058] LustreError: 93881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2539.755554] Lustre: server umount lustre-MDT0001 complete [ 2550.334244] Lustre: server umount lustre-OST0000 complete [ 2554.029826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2554.167558] LustreError: 98191:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2554.174015] LustreError: 98191:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2554.212652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2555.435911] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2558.095311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2558.332400] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2559.636492] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2561.865510] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2563.642779] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2567.662917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2567.732668] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2567.735028] LustreError: 98181:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2567.743274] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2567.745723] LustreError: 100272:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2567.747742] LustreError: 100272:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2567.749451] LustreError: 100272:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2567.772217] Lustre: server umount lustre-OST0000 complete [ 2567.773827] LustreError: 100272:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2570.409301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2570.467882] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2570.470764] LustreError: 98180:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2570.480632] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2570.483162] LustreError: 100738:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2570.485171] LustreError: 100738:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2570.487037] LustreError: 100738:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2570.506629] Lustre: server umount lustre-OST0000 complete [ 2570.507922] LustreError: 100738:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2573.561063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2573.621503] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2573.623307] Lustre: Found index 0 for lustre-OST0000, updating log [ 2573.625276] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2573.646203] Lustre: lustre-OST0000: new disk, initializing [ 2573.648473] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2573.651496] Lustre: 101266:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2573.742571] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2575.573321] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2576.710087] Lustre: Failing over lustre-OST0000 [ 2576.760152] Lustre: server umount lustre-OST0000 complete [ 2578.912621] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2578.912724] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2578.913131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2578.913135] Lustre: Skipped 2 previous similar messages [ 2578.915783] Lustre: Skipped 2 previous similar messages [ 2583.008395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2583.010525] Lustre: Skipped 2 previous similar messages [ 2588.128410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2588.130103] Lustre: Skipped 1 previous similar message [ 2592.223127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2592.291197] Lustre: server umount lustre-MDT0000 complete [ 2593.248972] LustreError: 98192:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2593.253697] LustreError: 98192:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2593.539236] LustreError: 98171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787995544 with bad export cookie 1431848556132156070 [ 2593.543376] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2593.544928] LustreError: 98171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2593.663591] Lustre: server umount lustre-MDT0001 complete [ 2596.500983] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 05:25:46 (1787995546) [ 2601.683309] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 2605.186568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2605.327937] LustreError: 103606:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2605.351155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2606.550273] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2609.195847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2610.680535] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2611.617641] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2613.781250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2615.951414] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2618.711649] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2618.743459] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2618.774368] Lustre: lustre-OST0001: new disk, initializing [ 2618.776137] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2618.778612] Lustre: 105923:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2620.298072] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2620.301625] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2620.340604] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2620.872110] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2623.869173] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2629.463158] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2632.027409] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.28@tcp (stopping) [ 2634.720384] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2634.723720] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2636.945276] Lustre: server umount lustre-MDT0001 complete [ 2637.139455] LustreError: 103602:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2640.493937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2640.540903] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2640.543461] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2640.560705] Lustre: 103594:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2640.832386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2640.834752] Lustre: Skipped 3 previous similar messages [ 2642.279123] LustreError: 107650:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2642.283057] LustreError: 107650:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2642.287217] LustreError: 103600:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2642.329160] LustreError: 107668:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.128@tcp: cfg command failed: rc = -17 [ 2642.332819] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2642.332819] [ 2642.393686] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2645.991114] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2646.139108] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 05:26:35 (1787995595) [ 2651.833187] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 2657.971810] Lustre: 109529:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2665.978202] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2667.140707] Lustre: 110662:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2712.543933] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2712.544037] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2712.544491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2712.544499] Lustre: Skipped 4 previous similar messages [ 2712.551939] Lustre: Skipped 5 previous similar messages [ 2717.331726] Lustre: server umount lustre-MDT0000 complete [ 2717.664523] LustreError: 103603:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2717.670277] LustreError: 103603:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2718.472691] LustreError: 106641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787995669 with bad export cookie 1431848556132156980 [ 2718.473778] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2718.476052] LustreError: 106641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2718.576307] Lustre: server umount lustre-MDT0001 complete [ 2729.992864] Lustre: server umount lustre-OST0000 complete [ 2741.595968] Lustre: server umount lustre-OST0001 complete [ 2743.326986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2746.418462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2749.166861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2752.322823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2758.256097] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 2761.566029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2761.572047] Lustre: lustre-MDT0000: reset Object Index mappings [ 2761.709163] LustreError: 114583:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2761.732211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2763.120963] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2766.091357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2766.099937] Lustre: lustre-MDT0001: reset Object Index mappings [ 2767.840482] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2768.881190] Lustre: 115731:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2771.407288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2772.710434] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 2773.833235] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2775.783878] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 2776.863478] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2779.150296] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2782.185878] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 2782.186669] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 2782.271502] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2783.401541] Lustre: 117606:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2796.777637] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 05:29:06 (1787995746) [ 2827.743989] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2827.747702] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2827.752814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2827.755132] Lustre: Skipped 3 previous similar messages [ 2832.100052] Lustre: server umount lustre-MDT0000 complete [ 2833.303223] LustreError: 114563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787995784 with bad export cookie 1431848556132203726 [ 2833.306954] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2833.307402] LustreError: 114563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2833.377079] LustreError: 114578:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2833.382371] LustreError: 114578:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2839.681824] Lustre: server umount lustre-MDT0001 complete [ 2855.391180] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2855.427791] Lustre: server umount lustre-OST0000 complete [ 2862.753665] Lustre: server umount lustre-OST0001 complete [ 2865.162127] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 2868.715866] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 2882.341210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2882.437013] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2882.447467] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2882.483315] Lustre: lustre-MDT0000: new disk, initializing [ 2882.520267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2883.861844] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2886.969530] Lustre: server umount lustre-MDT0000 complete [ 2890.249122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2890.370901] Lustre: Found index 0 for lustre-MDT0000, updating log [ 2890.375228] Lustre: 121863:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2890.382135] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 2890.458687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2890.462728] Lustre: Skipped 4 previous similar messages [ 2891.820646] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2893.428857] Lustre: server umount lustre-MDT0000 complete [ 2896.400525] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 05:30:46 (1787995846) [ 2896.912678] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 2897.503210] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 05:30:47 (1787995847) [ 2903.367523] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 2906.805675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2908.251335] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2911.120601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2911.162269] Lustre: 123911:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2911.185160] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2911.187065] Lustre: Skipped 1 previous similar message [ 2911.222389] Lustre: lustre-MDT0001: new disk, initializing [ 2911.266089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2911.269795] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2912.536757] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2914.816970] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2917.026384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2917.123185] Lustre: lustre-OST0000: new disk, initializing [ 2917.125093] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2917.128111] Lustre: 125489:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2917.673615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2917.676877] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2917.705749] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2919.049497] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2923.337459] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2923.401669] Lustre: 126359:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2925.054389] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2925.057293] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2925.093306] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2925.299764] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2929.537649] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2930.744385] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3040.246869] Lustre: 132728:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3040.249740] Lustre: 132728:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3040.251945] Lustre: 132728:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3040.254176] Lustre: 132728:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3040.256569] Lustre: 132728:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3040.258704] Lustre: 132728:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3122.160351] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 05:34:31 (1787996071) [ 3134.947888] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3134.951083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3134.954763] Lustre: Skipped 8 previous similar messages [ 3134.957232] Lustre: Skipped 10 previous similar messages [ 3139.836026] Lustre: server umount lustre-MDT0000 complete [ 3140.064237] LustreError: 127164:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3140.069045] LustreError: 127164:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 3141.245549] LustreError: 123905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787996092 with bad export cookie 1431848556132255813 [ 3141.251084] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3147.481341] Lustre: server umount lustre-MDT0001 complete [ 3163.103127] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3163.142608] Lustre: server umount lustre-OST0000 complete [ 3170.529923] Lustre: server umount lustre-OST0001 complete [ 3172.640284] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 3175.758672] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 3187.825984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3187.909539] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3187.919986] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3187.922065] Lustre: Skipped 1 previous similar message [ 3187.956996] Lustre: lustre-MDT0000: new disk, initializing [ 3187.958471] Lustre: Skipped 1 previous similar message [ 3187.976959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3187.979230] Lustre: Skipped 4 previous similar messages [ 3187.983980] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3189.302342] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3193.386275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3193.430742] Lustre: 135741:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3193.535961] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3194.812196] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3198.639651] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3200.724907] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3203.073191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3203.203652] Lustre: 137711:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3204.862624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3204.864868] Lustre: Skipped 1 previous similar message [ 3204.866398] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3204.913464] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3205.405222] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3209.233018] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3212.445723] Lustre: lustre-MDD0000: changelog on [ 3213.521042] Lustre: lustre-MDD0001: changelog on [ 3540.362230] Lustre: 136548:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 3571.464142] Lustre: lustre-MDD0001: changelog off [ 3572.570171] Lustre: lustre-MDD0000: changelog off [ 3578.851348] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3578.853241] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3578.854632] LustreError: Skipped 1 previous similar message [ 3578.854722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3578.854729] Lustre: Skipped 5 previous similar messages [ 3578.859369] Lustre: Skipped 3 previous similar messages [ 3580.637177] Lustre: server umount lustre-OST0000 complete [ 3582.944182] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3588.319184] Lustre: server umount lustre-MDT0000 complete [ 3589.089085] LustreError: 135753:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3589.093799] LustreError: 135753:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 3589.507275] LustreError: 135778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787996540 with bad export cookie 1431848556133053393 [ 3589.510707] LustreError: 135778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 3589.510833] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3589.627173] Lustre: server umount lustre-MDT0001 complete [ 3593.790749] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 3594.963747] Key type lgssc unregistered [ 3595.092493] LNet: 146481:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3595.095764] LNetError: 146481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3595.110515] LNet: Removed LNI 192.168.202.128@tcp [ 3595.459105] Key type .llcrypt unregistered [ 3595.460277] Key type ._llcrypt unregistered [ 3602.245439] Key type ._llcrypt registered [ 3602.246518] Key type .llcrypt registered [ 3602.288751] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 3608.630805] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 3609.125717] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3609.149863] alg: No test for adler32 (adler32-zlib) [ 3610.028215] Lustre: Lustre: Build Version: 2.17.57_44_gac22094 [ 3610.130508] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 3611.719152] Key type lgssc registered [ 3612.304130] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3624.723399] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3624.731034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3625.827279] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3625.837116] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3625.874807] Lustre: lustre-MDT0000: new disk, initializing [ 3625.896349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3625.901936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3627.327252] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3631.482644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3631.525378] Lustre: 149656:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3631.555496] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3631.557160] Lustre: Skipped 1 previous similar message [ 3631.597609] Lustre: lustre-MDT0001: new disk, initializing [ 3631.632974] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3631.647564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3631.651075] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3633.171405] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3637.159472] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3639.065384] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3641.858770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3641.994584] Lustre: lustre-OST0000: new disk, initializing [ 3641.998179] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3642.005612] Lustre: 151621:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3642.057738] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3644.038979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3644.042141] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3644.093501] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3644.352254] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3648.848820] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3651.092218] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_module obdecho/obdecho [ 3651.744727] LustreError: 152625:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 3651.747545] LustreError: 152625:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 3652.415569] LustreError: 152721:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 3657.184534] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3657.188054] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3657.192990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3659.746973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3659.749370] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3659.754589] Lustre: Skipped 1 previous similar message [ 3662.632740] Lustre: server umount lustre-MDT0000 complete [ 3663.856410] LustreError: 149649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787996614 with bad export cookie 2784948764034587481 [ 3663.861922] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3663.996777] Lustre: server umount lustre-MDT0001 complete [ 3675.930577] Lustre: server umount lustre-OST0000 complete [ 3683.765785] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 3686.995270] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 3700.431459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3700.521731] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3700.531844] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3700.571136] Lustre: lustre-MDT0000: new disk, initializing [ 3700.593467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3700.601028] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3702.189668] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3706.837699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3706.886953] Lustre: 156429:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3706.918765] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3706.921018] Lustre: Skipped 1 previous similar message [ 3706.965493] Lustre: lustre-MDT0001: new disk, initializing [ 3707.011573] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3707.024690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3707.028233] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3708.685022] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3712.955725] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3715.046806] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3718.200085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3718.351009] Lustre: lustre-OST0000: new disk, initializing [ 3718.353664] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3718.357403] Lustre: 158395:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3718.404494] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3720.697925] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3724.791994] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3726.624607] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3728.395668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3728.399881] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3728.445278] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3728.807395] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 3730.647975] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3730.735822] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3733.476754] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3733.481271] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3733.484432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3733.487767] Lustre: Skipped 1 previous similar message [ 3733.493327] Lustre: Skipped 3 previous similar messages [ 3737.704852] Lustre: server umount lustre-OST0000 complete [ 3743.200090] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3743.203656] LustreError: Skipped 1 previous similar message [ 3743.205491] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3743.211134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3745.632938] Lustre: server umount lustre-MDT0000 complete [ 3747.159525] LustreError: 156421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787996697 with bad export cookie 2784948764034588426 [ 3747.163040] LustreError: 156421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3747.164287] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3747.297584] Lustre: server umount lustre-MDT0001 complete [ 3748.666909] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 05:44:58 (1787996698) [ 3749.312658] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 3749.982266] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 05:44:59 (1787996699) [ 3753.055533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3753.256089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3754.564367] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3757.669607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3759.382823] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3762.114212] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3764.073005] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3766.575466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3768.831595] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3771.700842] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3787.610649] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.28@tcp (stopping) [ 3787.612628] Lustre: Skipped 1 previous similar message [ 3788.769258] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3788.779965] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3788.793171] Lustre: Skipped 1 previous similar message [ 3791.017213] Lustre: server umount lustre-MDT0000 complete [ 3792.385356] LustreError: 160798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787996743 with bad export cookie 2784948764034589294 [ 3792.388482] LustreError: 160798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3792.389341] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3792.723991] LustreError: 160842:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3792.731261] LustreError: 160842:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3794.400822] LustreError: 162817:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3797.408586] LustreError: 160814:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3797.409108] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3797.424347] Lustre: Skipped 6 previous similar messages [ 3798.647700] Lustre: server umount lustre-MDT0001 complete [ 3801.219498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3806.488155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3806.626456] LustreError: 164677:0:(llog.c:1655:llog_backup()) MGC192.168.202.128@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 3806.631130] Lustre: 164677:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.128@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 3823.072426] LustreError: 164682:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3823.081543] LustreError: 164682:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3828.192246] LustreError: 164684:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3828.198158] LustreError: 164684:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3837.408713] LustreError: 164705:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3837.415102] LustreError: 164705:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 3855.187858] LustreError: 164683:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3855.198856] LustreError: 164683:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 3867.615408] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 3867.617606] Lustre: 164677:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.128@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 3888.097501] LustreError: 164682:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3888.108980] LustreError: 164682:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 19 previous similar messages [ 3908.584153] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3908.586165] Lustre: Skipped 2 previous similar messages [ 3909.843543] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3912.438125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3912.671674] LustreError: 147881:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b239d271 x1874850281000448/t0(0) o250->MGC192.168.202.128@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3912.680195] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3914.052218] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3916.287761] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3916.760940] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3917.796791] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3918.514174] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3922.087473] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 3 sec [ 3927.626905] Lustre: server umount lustre-MDT0000 complete [ 3928.033778] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3928.039099] Lustre: Skipped 5 previous similar messages [ 3928.041552] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3929.173046] LustreError: 163619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787996879 with bad export cookie 2784948764034595734 [ 3929.179838] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3929.181537] LustreError: 163619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3933.153031] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3933.154771] Lustre: Skipped 1 previous similar message [ 3935.371919] Lustre: server umount lustre-MDT0001 complete [ 3937.052614] Lustre: server umount lustre-OST0000 complete [ 3943.904471] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 3946.958552] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 3958.512883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3958.609534] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3958.621505] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3958.659616] Lustre: lustre-MDT0000: new disk, initializing [ 3958.687308] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3960.026060] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3964.269082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3964.323115] Lustre: 169748:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3964.435423] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3965.948585] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3969.786078] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3971.725492] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3974.117696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3974.233500] Lustre: 171717:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3974.259038] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3974.261178] Lustre: Skipped 3 previous similar messages [ 3976.257474] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3978.120312] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3978.173397] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3980.248515] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3981.928293] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3981.993697] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3983.227296] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3983.301059] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3988.451340] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3988.451615] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3988.459261] Lustre: Skipped 2 previous similar messages [ 3988.459873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3988.463732] LustreError: Skipped 1 previous similar message [ 3988.471909] Lustre: Skipped 2 previous similar messages [ 3990.252104] Lustre: server umount lustre-OST0000 complete [ 3997.694718] Lustre: server umount lustre-MDT0000 complete [ 3998.689111] LustreError: 169758:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3998.696144] LustreError: 169758:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 23 previous similar messages [ 3999.226952] LustreError: 169740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787996949 with bad export cookie 2784948764034612534 [ 3999.231810] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3999.232950] LustreError: 169740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3999.372119] Lustre: server umount lustre-MDT0001 complete [ 4002.746109] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 05:49:12 (1787996952) [ 4005.287618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4007.069904] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4010.015227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4011.519109] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4014.117667] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4016.248620] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4018.511734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4020.694897] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4023.327125] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4038.414295] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 05:49:48 (1787996988) [ 4040.160275] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4040.164740] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4040.171626] Lustre: Skipped 2 previous similar messages [ 4045.544170] Lustre: server umount lustre-OST0000 complete [ 4053.103382] Lustre: server umount lustre-MDT0000 complete [ 4054.449480] LustreError: 177120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787997005 with bad export cookie 2784948764034613402 [ 4054.453007] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4054.453753] LustreError: 177120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4054.581181] Lustre: server umount lustre-MDT0001 complete [ 4058.481047] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 4059.450587] Key type lgssc unregistered [ 4059.573368] LNet: 177872:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4059.575655] LNetError: 177872:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4059.584314] LNet: Removed LNI 192.168.202.128@tcp [ 4059.917496] Key type .llcrypt unregistered [ 4059.919511] Key type ._llcrypt unregistered [ 4067.648076] Key type ._llcrypt registered [ 4067.649347] Key type .llcrypt registered [ 4067.687656] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 4068.032779] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4068.064571] alg: No test for adler32 (adler32-zlib) [ 4068.920324] Lustre: Lustre: Build Version: 2.17.57_44_gac22094 [ 4069.003562] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 4070.591230] Key type lgssc registered [ 4071.140407] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4074.469048] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4074.479608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4089.951354] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4095.071343] LustreError: 179091:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.128@tcp: failed processing log, type 4: rc = -110 [ 4100.192498] LustreError: 179091:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 4100.196274] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 4100.197926] LustreError: MGC192.168.202.128@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 4100.201061] LustreError: 179091:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 4100.204593] LustreError: 179091:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 4100.206762] LustreError: 179091:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 4100.225725] Lustre: server umount lustre-OST0000 complete [ 4100.227112] LustreError: 179091:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 4102.926039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4104.088147] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4105.426660] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4108.296177] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4108.417200] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4109.795391] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4112.054385] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4113.699615] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4116.227498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4116.324753] LustreError: 181406:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 4116.330714] Lustre: 181406:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.202.128@tcp: invalid local config log lustre-OST0000: rc = -22 [ 4116.387896] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4118.574370] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4121.632142] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4123.618116] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4123.619862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4123.623147] Lustre: Skipped 2 previous similar messages [ 4123.625522] Lustre: Skipped 1 previous similar message [ 4128.693909] Lustre: server umount lustre-MDT0000 complete [ 4128.738947] LustreError: 179592:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4128.747421] LustreError: 179592:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4130.281319] LustreError: 179573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787997081 with bad export cookie 14210501251062969799 [ 4130.285805] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4130.411440] Lustre: server umount lustre-MDT0001 complete [ 4141.975629] Lustre: server umount lustre-OST0000 complete [ 4145.435206] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 4148.536706] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 4162.322498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4162.402171] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4162.411396] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4162.444888] Lustre: lustre-MDT0000: new disk, initializing [ 4162.464827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4162.470204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4163.769669] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4167.772457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4167.812765] Lustre: 184924:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4167.836435] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4167.839134] Lustre: Skipped 1 previous similar message [ 4167.879048] Lustre: lustre-MDT0001: new disk, initializing [ 4167.911175] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4167.922952] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4167.926536] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4169.280098] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4173.053487] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4174.862178] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4177.474489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4177.573664] Lustre: lustre-OST0000: new disk, initializing [ 4177.576789] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4177.581106] Lustre: 186891:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4177.613329] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4178.366255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4178.370533] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4178.402848] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4179.634288] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4183.860632] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4185.441544] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4185.519474] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4186.952890] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4187.029241] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4188.642396] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4188.642882] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4188.644090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4188.644099] Lustre: Skipped 1 previous similar message [ 4188.645254] LustreError: Skipped 1 previous similar message [ 4188.650199] Lustre: Skipped 1 previous similar message [ 4193.764050] Lustre: server umount lustre-OST0000 complete [ 4198.880958] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4198.881227] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4198.881666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4198.881672] Lustre: Skipped 1 previous similar message [ 4198.884914] Lustre: Skipped 1 previous similar message [ 4201.334378] Lustre: server umount lustre-MDT0000 complete [ 4202.666797] LustreError: 184917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787997153 with bad export cookie 14210501251062970499 [ 4202.672514] LustreError: 184917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4202.673300] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4202.795076] Lustre: server umount lustre-MDT0001 complete [ 4205.802890] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 05:52:35 (1787997155) [ 4211.635832] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 4212.748922] Key type lgssc unregistered [ 4212.885579] LNet: 189834:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4212.889517] LNetError: 189834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4212.901633] LNet: Removed LNI 192.168.202.128@tcp [ 4213.263368] Key type .llcrypt unregistered [ 4213.264337] Key type ._llcrypt unregistered [ 4220.642766] Key type ._llcrypt registered [ 4220.643988] Key type .llcrypt registered [ 4220.690580] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 4221.050465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4221.103039] alg: No test for adler32 (adler32-zlib) [ 4221.975647] Lustre: Lustre: Build Version: 2.17.57_44_gac22094 [ 4222.085405] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 4223.679133] Key type lgssc registered [ 4224.113454] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4226.659716] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4226.666116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4227.828838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4229.207634] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4232.058270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4232.180021] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4233.486895] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4235.874951] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4237.811467] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4240.405582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4240.488915] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4240.490748] LustreError: 192891:0:(llog.c:1696:llog_backup()) MGC192.168.202.128@tcp: failed to backup log lustre-OST0000: rc = -28 [ 4240.494467] Lustre: 192891:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.202.128@tcp: can't backup local config lustre-OST0000: rc = -28 [ 4240.539495] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4242.522832] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4245.219426] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4247.011652] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4247.013428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4247.016685] Lustre: Skipped 2 previous similar messages [ 4247.019904] Lustre: Skipped 2 previous similar messages [ 4252.130153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4252.133873] Lustre: Skipped 2 previous similar messages [ 4257.251311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4257.255191] Lustre: Skipped 2 previous similar messages [ 4260.319189] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4260.428810] Lustre: server umount lustre-MDT0000 complete [ 4261.794511] LustreError: 191055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787997212 with bad export cookie 12168324296547165562 [ 4261.798170] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4261.801571] LustreError: 191055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4261.922734] Lustre: server umount lustre-MDT0001 complete [ 4273.625259] Lustre: server umount lustre-OST0000 complete [ 4276.694404] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 4279.820044] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 4292.246508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4292.330987] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4292.340820] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4292.377277] Lustre: lustre-MDT0000: new disk, initializing [ 4292.398944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4292.404536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4293.775534] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4297.950151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4297.996873] Lustre: 196409:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4298.018956] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4298.020255] Lustre: Skipped 1 previous similar message [ 4298.021473] LustreError: 197194:0:(llog.c:1696:llog_backup()) MGC192.168.202.128@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 4298.023955] LustreError: 197194:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 4298.026506] Lustre: 197194:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.128@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 4298.035562] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4298.037807] Lustre: Skipped 1 previous similar message [ 4298.081413] Lustre: lustre-MDT0001: new disk, initializing [ 4298.121330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4298.133777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4298.138044] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4299.548619] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4303.324481] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4305.229130] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4307.487174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4307.596280] Lustre: *** cfs_fail_loc=131e, val=0*** [ 4307.597821] Lustre: Skipped 1 previous similar message [ 4307.599678] LustreError: 198356:0:(llog.c:1696:llog_backup()) MGC192.168.202.128@tcp: failed to backup log lustre-OST0000: rc = -28 [ 4307.603245] LustreError: 198356:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 4307.606254] Lustre: 198356:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.128@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 4307.611721] Lustre: 198356:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 4307.623186] Lustre: lustre-OST0000: new disk, initializing [ 4307.625561] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4307.628529] Lustre: 198379:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4307.662094] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4309.759497] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4310.907375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4310.911445] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4310.948051] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4313.350440] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4314.695933] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4314.766244] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4316.126989] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4316.200264] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4321.251904] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4321.252094] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4321.254579] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4321.255577] Lustre: Skipped 1 previous similar message [ 4321.257531] LustreError: Skipped 1 previous similar message [ 4322.963428] Lustre: server umount lustre-OST0000 complete [ 4326.368808] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4326.369507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4326.372175] Lustre: Skipped 1 previous similar message [ 4326.373644] Lustre: Skipped 1 previous similar message [ 4330.334509] Lustre: server umount lustre-MDT0000 complete [ 4331.488088] LustreError: 196422:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4331.493944] LustreError: 196422:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4331.558259] LustreError: 197208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787997282 with bad export cookie 12168324296547166248 [ 4331.561533] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4331.561977] LustreError: 197208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4331.679575] Lustre: server umount lustre-MDT0001 complete [ 4334.393966] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 05:54:44 (1787997284) [ 4339.345593] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 4342.737247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4342.862123] LustreError: 201501:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4342.875024] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4344.133959] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4347.133114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4348.568239] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4349.545528] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4351.779274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4353.932674] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4356.971637] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4357.080728] Lustre: lustre-OST0001: new disk, initializing [ 4357.083404] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4357.086956] Lustre: 203808:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4359.161246] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4362.225406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4362.229403] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4362.245311] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4364.596844] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4368.128396] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4368.883594] Lustre: DEBUG MARKER: ADD OST3 [ 4370.392392] loop: module loaded [ 4370.416800] loop0: detected capacity change from 0 to 204800000 [ 4370.486012] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4370.489220] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4370.492686] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4372.383475] loop0: detected capacity change from 0 to 204800000 [ 4374.193964] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4374.275467] Lustre: lustre-OST0003: new disk, initializing [ 4374.277402] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 4374.279817] Lustre: 205539:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4375.443855] Lustre: DEBUG MARKER: STOP OST3 [ 4375.582348] Lustre: *** cfs_fail_loc=2109, val=2*** [ 4375.620809] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 4375.623854] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 4375.636777] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 4376.136797] Lustre: Failing over lustre-OST0003 [ 4376.191320] Lustre: server umount lustre-OST0003 complete [ 4376.426302] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4377.631072] LustreError: 205792:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 4380.429291] Lustre: DEBUG MARKER: START OST3 again [ 4380.648151] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4380.649337] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4380.654404] LustreError: 202983:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4380.664405] LustreError: 202983:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4382.668642] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4382.763346] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 4382.765542] Lustre: Skipped 4 previous similar messages [ 4382.772352] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 4384.098165] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4384.540200] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4384.540530] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4384.606710] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4397.963477] Lustre: Failing over lustre-OST0003 [ 4397.999409] Lustre: server umount lustre-OST0003 complete [ 4398.056776] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4398.060216] Lustre: Skipped 1 previous similar message [ 4398.062284] LustreError: 202982:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4398.066657] LustreError: 202982:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4403.181491] LustreError: 202983:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4403.192354] LustreError: 202983:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4413.412818] LustreError: 202984:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4413.413996] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4413.415624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4413.415631] Lustre: Skipped 1 previous similar message [ 4413.419479] LustreError: 202984:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4413.424677] Lustre: Skipped 3 previous similar messages [ 4416.610370] Lustre: server umount lustre-MDT0000 complete [ 4417.819448] LustreError: 201527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787997368 with bad export cookie 12168324296547167116 [ 4417.819740] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4417.823028] LustreError: 201527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4432.351087] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4432.441958] Lustre: server umount lustre-MDT0001 complete [ 4433.764750] Lustre: server umount lustre-OST0000 complete [ 4435.031573] Lustre: server umount lustre-OST0001 complete [ 4437.181441] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 4440.302855] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 4452.832950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4452.939080] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4452.949411] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4452.988464] Lustre: lustre-MDT0000: new disk, initializing [ 4453.014186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4453.021560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4454.435723] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4459.512705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4459.569405] Lustre: 210949:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4459.727672] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4461.248328] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4465.430480] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4467.701719] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4470.207705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4470.347914] Lustre: lustre-OST0000: new disk, initializing [ 4470.349485] Lustre: Skipped 1 previous similar message [ 4470.351243] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4470.353166] Lustre: Skipped 2 previous similar messages [ 4470.355392] Lustre: 212916:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4471.744508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4471.747836] Lustre: Skipped 1 previous similar message [ 4471.750034] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4471.790585] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4472.536596] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4476.463629] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4477.911503] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4477.978727] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4479.266165] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4479.344825] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4482.023129] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4482.023145] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4482.024123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4482.024131] Lustre: Skipped 9 previous similar messages [ 4482.032204] Lustre: Skipped 4 previous similar messages [ 4482.037103] LustreError: Skipped 1 previous similar message [ 4494.303202] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4494.361396] Lustre: server umount lustre-OST0000 complete [ 4495.839957] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4501.796829] Lustre: server umount lustre-MDT0000 complete [ 4502.496634] LustreError: 211758:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4502.502251] LustreError: 211758:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4503.130409] LustreError: 210985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787997453 with bad export cookie 12168324296547169398 [ 4503.133655] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4503.135100] LustreError: 210985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4503.256985] Lustre: server umount lustre-MDT0001 complete [ 4504.435953] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 05:57:34 (1787997454) [ 4510.010414] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 4513.459293] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 4525.210774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4525.311680] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4525.323319] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4525.361185] Lustre: lustre-MDT0000: new disk, initializing [ 4525.386956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4526.709862] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4530.909059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4530.950650] Lustre: 217521:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4531.062583] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4532.484049] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4536.344265] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4538.409515] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4541.088552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4541.224061] Lustre: 219489:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4542.461393] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4543.410711] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4547.331311] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4548.816238] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4548.886973] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4550.254792] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4550.325075] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4552.673044] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4552.674649] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4552.677118] LustreError: Skipped 1 previous similar message [ 4552.681353] Lustre: Skipped 2 previous similar messages [ 4552.682742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4552.688502] Lustre: Skipped 9 previous similar messages [ 4557.095948] Lustre: server umount lustre-OST0000 complete [ 4561.888164] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4564.637518] Lustre: server umount lustre-MDT0000 complete [ 4565.972167] LustreError: 217511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787997516 with bad export cookie 12168324296547170280 [ 4565.975283] LustreError: 217511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4565.976127] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4566.108840] Lustre: server umount lustre-MDT0001 complete [ 4568.919152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4569.080503] LustreError: 221700:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4570.451697] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4573.329132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4574.927286] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4577.631985] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4579.949363] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4582.856922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4583.019911] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4583.021879] Lustre: Skipped 7 previous similar messages [ 4584.924787] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4587.625583] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4599.776042] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4605.675643] Lustre: server umount lustre-OST0000 complete [ 4613.467703] Lustre: server umount lustre-MDT0000 complete [ 4614.719308] LustreError: 221680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787997565 with bad export cookie 12168324296547171162 [ 4614.722386] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4614.722767] LustreError: 221680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4614.839927] Lustre: server umount lustre-MDT0001 complete [ 4721.415080] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 4722.548154] Key type lgssc unregistered [ 4722.679603] LNet: 225394:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4722.683498] LNetError: 225394:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4722.697582] LNet: Removed LNI 192.168.202.128@tcp [ 4723.144117] Key type .llcrypt unregistered [ 4723.145176] Key type ._llcrypt unregistered [ 4731.529965] Key type ._llcrypt registered [ 4731.530885] Key type .llcrypt registered [ 4731.568205] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 4737.640173] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 4738.215456] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4738.233441] alg: No test for adler32 (adler32-zlib) [ 4739.101708] Lustre: Lustre: Build Version: 2.17.57_44_gac22094 [ 4739.195631] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 4740.775210] Key type lgssc registered [ 4741.248972] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4753.204489] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4753.209802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4754.301584] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4754.313832] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4754.350454] Lustre: lustre-MDT0000: new disk, initializing [ 4754.374622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4754.381040] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4755.859771] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4760.055316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4760.099939] Lustre: 228719:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4760.126669] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4760.128386] Lustre: Skipped 1 previous similar message [ 4760.167796] Lustre: lustre-MDT0001: new disk, initializing [ 4760.204672] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4760.219412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4760.223407] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4761.539393] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4765.327835] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4767.093281] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4769.367871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4769.489975] Lustre: lustre-OST0000: new disk, initializing [ 4769.491838] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4769.495686] Lustre: 230682:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4769.497621] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 4769.552761] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4771.594903] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4774.895222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4774.898659] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4774.911075] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4775.439362] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4776.940796] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4777.022342] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4778.435727] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4778.521471] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4780.005678] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4780.006272] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4780.007332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4780.007780] LustreError: Skipped 1 previous similar message [ 4780.011230] Lustre: Skipped 1 previous similar message [ 4781.271306] Lustre: server umount lustre-OST0000 complete [ 4785.121495] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4785.122812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4785.125554] Lustre: Skipped 1 previous similar message [ 4785.127672] Lustre: Skipped 1 previous similar message [ 4788.634221] Lustre: server umount lustre-MDT0000 complete [ 4789.824459] LustreError: 229517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787997740 with bad export cookie 1480083839308909127 [ 4789.828052] LustreError: 229517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4789.828258] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4789.959771] Lustre: server umount lustre-MDT0001 complete [ 4796.895892] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 4802.457383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4802.615146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4804.050956] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4806.854452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4806.984525] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4808.456243] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4809.455430] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4811.626231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4813.731634] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4816.709103] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4816.790066] Lustre: lustre-OST0001: new disk, initializing [ 4816.792680] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4816.795788] Lustre: 235917:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4816.837787] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4816.840308] Lustre: Skipped 1 previous similar message [ 4818.847137] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4818.950870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4818.954283] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4818.997394] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4824.297938] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4831.009878] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4834.734323] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 06:03:04 (1787997784) [ 4837.856042] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4837.858579] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4837.862755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4841.563604] Lustre: server umount lustre-MDT0000 complete [ 4842.774346] LustreError: 233595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787997793 with bad export cookie 1480083839308909967 [ 4842.776951] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4842.777878] LustreError: 233595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4842.890384] Lustre: server umount lustre-MDT0001 complete [ 4853.698540] Lustre: server umount lustre-OST0000 complete [ 4865.192551] Lustre: server umount lustre-OST0001 complete [ 4867.321262] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 4870.403959] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 4882.139675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4882.224041] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4882.233675] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4882.271092] Lustre: lustre-MDT0000: new disk, initializing [ 4882.293180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4882.299392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4883.658035] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4887.750951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4887.796590] Lustre: 239859:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4887.913721] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4889.316230] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4893.036146] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4894.893632] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4897.261197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4897.384977] Lustre: lustre-OST0000: new disk, initializing [ 4897.386485] Lustre: Skipped 1 previous similar message [ 4897.388262] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4897.390231] Lustre: Skipped 2 previous similar messages [ 4897.392582] Lustre: 241825:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4898.616337] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4898.619876] Lustre: Skipped 1 previous similar message [ 4898.622386] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4898.656531] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4899.483833] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4903.151801] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4904.535556] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4904.605060] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4905.975852] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4906.068735] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4908.823372] Lustre: server umount lustre-OST0000 complete [ 4913.631866] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4913.634721] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4913.638896] Lustre: Skipped 3 previous similar messages [ 4913.640743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4913.642981] Lustre: Skipped 4 previous similar messages [ 4924.383109] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4924.383152] Lustre: 226947:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787997859/real 1787997859] req@ffff939310f75880 x1874851464850176/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787997875 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 4924.385211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4924.385218] Lustre: Skipped 4 previous similar messages [ 4924.400795] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4924.405632] Lustre: Skipped 1 previous similar message [ 4924.469625] Lustre: server umount lustre-MDT0000 complete [ 4924.895088] Lustre: 226948:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787997859/real 1787997859] req@ffff939313641f80 x1874851464849408/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787997875 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp-pre-0-1.0' uid:0 gid:0 projid:4294967295 [ 4925.808546] LustreError: 241824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787997876 with bad export cookie 1480083839308912060 [ 4925.811850] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4925.813744] LustreError: 241824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4925.935943] Lustre: server umount lustre-MDT0001 complete [ 4928.943932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4929.105904] LustreError: 244040:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4929.111971] LustreError: 244040:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4929.133944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4929.135964] Lustre: Skipped 2 previous similar messages [ 4930.567966] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4933.481797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4935.070206] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4937.547465] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4939.399725] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4941.941427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4944.038329] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4946.463968] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5091.813203] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5091.813845] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5091.814224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5091.814230] Lustre: Skipped 1 previous similar message [ 5091.817331] Lustre: Skipped 1 previous similar message [ 5097.106282] Lustre: server umount lustre-OST0000 complete [ 5104.596273] Lustre: server umount lustre-MDT0000 complete [ 5105.884477] LustreError: 244774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787998056 with bad export cookie 1480083839308912942 [ 5105.887897] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5105.888093] LustreError: 244774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5106.015093] Lustre: server umount lustre-MDT0001 complete [ 5212.435532] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 5213.474850] Key type lgssc unregistered [ 5213.615533] LNet: 247646:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5213.618206] LNetError: 247646:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5213.627480] LNet: Removed LNI 192.168.202.128@tcp [ 5213.975141] Key type .llcrypt unregistered [ 5213.976306] Key type ._llcrypt unregistered [ 5222.501985] Key type ._llcrypt registered [ 5222.502909] Key type .llcrypt registered [ 5222.539384] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 5229.088796] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 5229.520917] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5229.556369] alg: No test for adler32 (adler32-zlib) [ 5230.402635] Lustre: Lustre: Build Version: 2.17.57_44_gac22094 [ 5230.495377] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 5232.079158] Key type lgssc registered [ 5232.394363] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5245.280213] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 5248.453283] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5248.458113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5249.532933] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5249.541082] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5249.572898] Lustre: lustre-MDT0000: new disk, initializing [ 5249.590986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5249.596332] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5250.879449] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5254.806653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5254.841765] Lustre: 251686:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5254.861759] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5254.863345] Lustre: Skipped 1 previous similar message [ 5254.894400] Lustre: lustre-MDT0001: new disk, initializing [ 5254.921935] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5254.931795] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5254.935275] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5256.312455] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5258.611527] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5260.797017] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5260.899949] Lustre: lustre-OST0000: new disk, initializing [ 5260.902737] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5260.905210] Lustre: 253320:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5260.937390] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5262.889124] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5265.404167] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5265.407305] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5265.445123] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5266.960342] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5267.027380] Lustre: lustre-OST0001: new disk, initializing [ 5267.029508] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5267.032252] Lustre: 254191:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5267.065215] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5269.017152] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5274.395610] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5275.136567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5275.139580] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5275.172815] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5277.999102] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5285.346988] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5285.348654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5285.353332] Lustre: Skipped 2 previous similar messages [ 5285.356097] Lustre: Skipped 3 previous similar messages [ 5287.012703] Lustre: server umount lustre-MDT0000 complete [ 5288.230444] LustreError: 251679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787998238 with bad export cookie 18314284609761293591 [ 5288.234877] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5288.236185] LustreError: 251679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5288.368097] Lustre: server umount lustre-MDT0001 complete [ 5299.659793] Lustre: server umount lustre-OST0000 complete [ 5311.262377] Lustre: server umount lustre-OST0001 complete [ 5315.938560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5319.812089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5324.964537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5328.096313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 5334.531356] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 5337.937503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5337.945220] Lustre: lustre-MDT0000: reset Object Index mappings [ 5338.087697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5339.464240] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5342.413770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5342.418991] Lustre: lustre-MDT0001: reset Object Index mappings [ 5343.952197] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5344.973968] Lustre: 260331:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5347.246341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5347.360316] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5347.362763] Lustre: Skipped 1 previous similar message [ 5348.386155] LustreError: 260684:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5348.389275] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5348.392266] LustreError: 260684:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5349.227409] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5351.853777] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5353.002180] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 5353.667713] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5356.658695] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5357.828695] Lustre: 262198:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5363.170812] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5363.173234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5363.173674] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5363.174304] Lustre: Skipped 4 previous similar messages [ 5363.177020] Lustre: Skipped 2 previous similar messages [ 5365.013037] Lustre: server umount lustre-MDT0000 complete [ 5366.149204] LustreError: 259172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787998316 with bad export cookie 18314284609761300367 [ 5366.152679] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5366.153350] LustreError: 259172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5366.263987] Lustre: server umount lustre-MDT0001 complete [ 5377.546815] Lustre: server umount lustre-OST0000 complete [ 5389.743159] Lustre: server umount lustre-OST0001 complete [ 5397.139130] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 06:12:26 (1787998346) [ 5403.353971] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 5407.591407] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 5422.731477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5422.821574] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5422.831944] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5422.863345] Lustre: lustre-MDT0000: new disk, initializing [ 5422.882759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5422.884898] Lustre: Skipped 1 previous similar message [ 5422.889974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5424.363418] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5428.803017] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5428.844380] Lustre: 266447:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5428.969949] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5430.480819] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5434.325949] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5436.167639] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5438.485221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5438.600635] Lustre: lustre-OST0000: new disk, initializing [ 5438.602415] Lustre: Skipped 1 previous similar message [ 5438.604242] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5438.606286] Lustre: Skipped 2 previous similar messages [ 5438.608554] Lustre: 268416:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5439.738575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5439.741987] Lustre: Skipped 1 previous similar message [ 5439.743874] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5439.774511] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5440.522826] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5444.171131] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5445.563553] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5445.634996] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5446.994711] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5447.069136] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5450.208754] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5450.210152] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5450.211634] LustreError: Skipped 1 previous similar message [ 5450.216661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5450.219752] Lustre: Skipped 1 previous similar message [ 5455.329368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5455.331291] Lustre: Skipped 1 previous similar message [ 5460.449845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5460.451881] Lustre: Skipped 1 previous similar message [ 5461.983128] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5462.042261] Lustre: server umount lustre-OST0000 complete [ 5465.056056] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5465.059642] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5465.064592] Lustre: Skipped 1 previous similar message [ 5469.332805] Lustre: server umount lustre-MDT0000 complete [ 5470.481036] LustreError: 267254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787998421 with bad export cookie 18314284609761302439 [ 5470.483971] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5470.484665] LustreError: 267254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5470.593662] Lustre: server umount lustre-MDT0001 complete [ 5476.063518] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 5479.592120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5479.728973] LustreError: 271344:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5479.742403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5479.745196] Lustre: Skipped 2 previous similar messages [ 5481.155785] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5483.994233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5485.389509] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5486.307653] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5488.269873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5490.130397] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5492.953984] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5493.018469] Lustre: lustre-OST0001: new disk, initializing [ 5493.020138] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5493.023026] Lustre: 273651:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5494.869687] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5494.908287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5494.912023] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5494.944743] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5499.198230] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5500.382874] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 5502.126850] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 5502.136160] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 5504.480195] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5504.482715] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5504.485922] Lustre: Skipped 1 previous similar message [ 5504.488044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5504.490030] Lustre: Skipped 2 previous similar messages [ 5509.851412] Lustre: server umount lustre-MDT0000 complete [ 5510.626435] LustreError: 273015:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5510.632471] LustreError: 273015:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5511.046259] LustreError: 271326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787998461 with bad export cookie 18314284609761303321 [ 5511.048694] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5511.050083] LustreError: 271326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5511.162925] Lustre: server umount lustre-MDT0001 complete [ 5522.955046] Lustre: server umount lustre-OST0000 complete [ 5533.470565] Lustre: server umount lustre-OST0001 complete [ 5538.763218] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 5542.116739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5542.240839] LustreError: 276700:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5543.603865] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5546.575087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5546.723268] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5546.725602] Lustre: Skipped 4 previous similar messages [ 5548.244522] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5549.299130] Lustre: 277844:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 5551.615958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5552.806682] LustreError: 278197:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5552.817106] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 5553.853417] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5556.680146] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5557.806077] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 5558.701237] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5561.872498] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5563.100524] Lustre: 279715:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 5564.978635] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 5564.984711] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 5568.273849] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 06:15:18 (1787998518) [ 5582.815934] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5582.818229] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5582.821494] Lustre: Skipped 3 previous similar messages [ 5582.823270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5582.824972] Lustre: Skipped 4 previous similar messages [ 5585.887956] Lustre: server umount lustre-MDT0000 complete [ 5587.139332] LustreError: 276683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787998537 with bad export cookie 18314284609761309635 [ 5587.142287] LustreError: 276683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5587.142903] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5588.448469] LustreError: 276695:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5588.453230] LustreError: 276695:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5593.327368] Lustre: server umount lustre-MDT0001 complete [ 5600.637454] Lustre: server umount lustre-OST0000 complete [ 5616.095152] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5616.160394] Lustre: server umount lustre-OST0001 complete [ 5618.302855] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 5621.483543] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 5636.681476] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 5640.272967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5640.356303] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5640.366798] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5640.404098] Lustre: lustre-MDT0000: new disk, initializing [ 5640.432512] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5641.786456] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5645.815147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5645.854919] Lustre: 283962:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5645.954669] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5647.264990] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5651.646044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5651.749584] Lustre: 285595:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5653.023400] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5653.715258] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5657.901603] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5659.889680] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5660.012602] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5664.188361] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5668.932271] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 06:16:58 (1787998618) [ 5670.371162] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5670.372063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5670.376386] Lustre: Skipped 6 previous similar messages [ 5670.377765] Lustre: Skipped 8 previous similar messages [ 5675.673117] Lustre: server umount lustre-MDT0000 complete [ 5676.833105] LustreError: 284757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787998627 with bad export cookie 18314284609761317503 [ 5676.836317] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5676.837247] LustreError: 284757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5676.940289] Lustre: server umount lustre-MDT0001 complete [ 5687.816180] Lustre: server umount lustre-OST0000 complete [ 5699.217514] Lustre: server umount lustre-OST0001 complete [ 5701.831649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5701.976202] LustreError: 288707:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5701.984217] LustreError: 288707:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5702.008466] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5702.011203] Lustre: Skipped 6 previous similar messages [ 5703.287347] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5705.914933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5707.219217] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5709.380048] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5711.069649] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5713.135485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5715.013760] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5717.251926] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5737.812103] LustreError: 290955:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5737.818196] LustreError: 290955:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 5738.442871] loop1: detected capacity change from 0 to 204800000 [ 5738.516306] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5738.520072] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5738.523952] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5740.086967] loop1: detected capacity change from 0 to 204800000 [ 5741.792737] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5741.818523] LustreError: 288695:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs32400: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 5741.824663] LustreError: 288695:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs32400 [ 5741.827422] LustreError: 288695:0:(mgs_handler.c:517:mgs_target_reg()) fs32400-MDT0001: Error checking target index: rc = -13 [ 5741.831731] LustreError: fs32400-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 5741.834625] LustreError: 292450:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 5741.837304] LustreError: 292450:0:(tgt_mount.c:2129:server_put_super()) no obd fs32400-MDT0001 [ 5741.839647] LustreError: 292450:0:(tgt_mount.c:116:server_deregister_mount()) fs32400-MDT0001 not registered [ 5741.850658] Lustre: server umount fs32400-MDT0001 complete [ 5741.851790] LustreError: 292450:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 5744.373994] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5744.400932] Lustre: Setting parameter fs32400-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs32400-MDT0001 [ 5744.404405] Lustre: Skipped 2 previous similar messages [ 5744.418496] Lustre: srv-fs32400-MDT0001: No data found on store. Initialize space. [ 5744.421133] Lustre: Skipped 4 previous similar messages [ 5744.457467] Lustre: fs32400-MDT0001: new disk, initializing [ 5744.459420] Lustre: Skipped 3 previous similar messages [ 5744.497329] Lustre: srv-fs32400-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 5744.500546] Lustre: 292914:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs32400-MDT0001-mdtlov: can't create new llog: rc = -115 [ 5744.504296] LustreError: 292914:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs32400-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 5745.763399] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5748.577590] Lustre: server umount fs32400-MDT0001 complete [ 5752.678750] loop1: detected capacity change from 0 to 204800000 [ 5754.512794] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5754.621731] Lustre: 294478:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs32400-MDT0001-mdtlov: can't create new llog: rc = -115 [ 5754.625747] LustreError: 294478:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs32400-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 5755.871377] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5757.333798] Lustre: server umount fs32400-MDT0001 complete [ 5762.528051] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5762.530342] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5762.533795] Lustre: Skipped 2 previous similar messages [ 5762.535681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5762.537456] Lustre: Skipped 7 previous similar messages [ 5766.491313] Lustre: server umount lustre-MDT0000 complete [ 5771.634342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5771.676504] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5773.098674] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5776.034412] loop1: detected capacity change from 0 to 204800000 [ 5776.870246] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5778.071448] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5778.194548] Lustre: 296919:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs32400-MDT0001-mdtlov: can't create new llog: rc = -115 [ 5778.197319] LustreError: 296919:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs32400-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 5779.545435] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5781.178016] Lustre: server umount fs32400-MDT0001 complete [ 5792.227883] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5794.194710] Lustre: server umount lustre-OST0000 complete [ 5801.624553] Lustre: server umount lustre-MDT0000 complete [ 5802.464516] LustreError: 290560:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5802.470153] LustreError: 290560:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 36 previous similar messages [ 5802.930279] LustreError: 288689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787998753 with bad export cookie 18314284609761320184 [ 5802.933893] LustreError: 288689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5803.050719] Lustre: server umount lustre-MDT0001 complete [ 5806.695608] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 5807.678888] Key type lgssc unregistered [ 5807.805390] LNet: 299262:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5807.807652] LNetError: 299262:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5807.816244] LNet: Removed LNI 192.168.202.128@tcp [ 5808.121143] Key type .llcrypt unregistered [ 5808.122333] Key type ._llcrypt unregistered [ 5815.504619] Key type ._llcrypt registered [ 5815.505633] Key type .llcrypt registered [ 5815.543798] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 5815.995149] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5816.005101] alg: No test for adler32 (adler32-zlib) [ 5816.856485] Lustre: Lustre: Build Version: 2.17.57_44_gac22094 [ 5816.942860] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 5818.519112] Key type lgssc registered [ 5818.860404] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5821.352763] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5821.358142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5822.494187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5823.722876] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5826.391365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5826.532131] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5827.866732] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5830.107772] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5831.872772] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5834.135020] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5834.256808] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5836.305949] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5836.324402] LustreError: 302705:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5836.330784] LustreError: 302705:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5838.677473] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5841.379284] LustreError: 302338:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5846.499185] LustreError: 302705:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5849.434869] LustreError: 302340:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5849.442325] LustreError: 302340:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5854.548374] LustreError: 302339:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5854.555023] LustreError: 302339:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5855.371815] loop1: detected capacity change from 0 to 204800000 [ 5855.435978] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5855.440313] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5855.444693] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5857.281853] loop1: detected capacity change from 0 to 204800000 [ 5859.442184] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5859.495428] Lustre: fs2757-OST0000: new disk, initializing [ 5859.497415] Lustre: srv-fs2757-OST0000: No data found on store. Initialize space. [ 5859.499456] Lustre: 304264:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5859.522656] Lustre: fs2757-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5861.659488] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5862.882342] LustreError: 302339:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5862.888299] LustreError: 302339:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 5865.274963] Lustre: server umount fs2757-OST0000 complete [ 5869.432181] loop1: detected capacity change from 0 to 204800000 [ 5871.331540] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5871.394316] Lustre: fs2757-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5873.257847] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5876.325105] loop2: detected capacity change from 0 to 204800000 [ 5876.410436] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5876.415582] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5876.421192] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 5878.407074] loop2: detected capacity change from 0 to 204800000 [ 5880.149933] LustreError: 302706:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5880.157869] LustreError: 302706:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 5880.473549] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5880.507981] LustreError: fs2757-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 5880.513580] LustreError: fs2757-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 5880.517894] LustreError: 307299:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 5880.521190] LustreError: 307299:0:(tgt_mount.c:2129:server_put_super()) no obd fs2757-OST0001 [ 5880.523958] LustreError: 307299:0:(tgt_mount.c:116:server_deregister_mount()) fs2757-OST0001 not registered [ 5880.538368] Lustre: server umount fs2757-OST0001 complete [ 5880.540267] LustreError: 307299:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 5884.447305] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5884.510117] Lustre: fs2757-OST0001: new disk, initializing [ 5884.512451] Lustre: srv-fs2757-OST0001: No data found on store. Initialize space. [ 5884.515781] Lustre: 307850:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5884.547843] Lustre: fs2757-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5887.380788] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5892.026425] Lustre: server umount fs2757-OST0000 complete [ 5896.463515] Lustre: server umount fs2757-OST0001 complete [ 5901.349254] loop1: detected capacity change from 0 to 204800000 [ 5903.618974] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5903.697856] Lustre: fs2757-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5906.165943] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5908.623950] loop2: detected capacity change from 0 to 204800000 [ 5910.762343] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5913.139768] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5914.082346] LustreError: 302340:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5914.089135] LustreError: 302340:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 19 previous similar messages [ 5917.192993] Lustre: server umount fs2757-OST0000 complete [ 5920.918828] Lustre: server umount fs2757-OST0001 complete [ 5924.331474] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [ 5928.417649] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5928.422982] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5928.428046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5929.443094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5929.447152] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5929.463387] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5934.561510] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [ 5934.561545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5934.567089] Lustre: Skipped 2 previous similar messages [ 5939.681439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5940.191145] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5940.259764] Lustre: server umount lustre-OST0000 complete [ 5944.288220] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5944.291530] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5944.298130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5944.299804] Lustre: Skipped 1 previous similar message [ 5947.995950] Lustre: server umount lustre-MDT0000 complete [ 5949.297215] LustreError: 300491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787998900 with bad export cookie 8777735758056569154 [ 5949.301223] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5949.301808] LustreError: 300491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5949.439167] Lustre: server umount lustre-MDT0001 complete [ 5953.547967] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 5954.926365] Key type lgssc unregistered [ 5955.064327] LNet: 313764:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5955.067727] LNetError: 313764:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5955.077444] LNet: Removed LNI 192.168.202.128@tcp [ 5955.461385] Key type .llcrypt unregistered [ 5955.463409] Key type ._llcrypt unregistered [ 5971.702387] Key type ._llcrypt registered [ 5971.703584] Key type .llcrypt registered [ 5971.751113] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 5979.410632] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 5979.940567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5979.963252] alg: No test for adler32 (adler32-zlib) [ 5980.851078] Lustre: Lustre: Build Version: 2.17.57_44_gac22094 [ 5980.959857] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 5982.559095] Key type lgssc registered [ 5982.921529] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6007.134948] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6007.151954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6008.243104] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6008.253399] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6008.286467] Lustre: lustre-MDT0000: new disk, initializing [ 6008.308179] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6008.313764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6009.941585] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6019.094842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6019.187647] Lustre: 317597:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6019.226192] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6019.231738] Lustre: Skipped 1 previous similar message [ 6019.301870] Lustre: lustre-MDT0001: new disk, initializing [ 6019.381706] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6019.408456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6019.412155] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6025.698588] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6035.879534] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6041.014541] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6046.270445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6046.425946] Lustre: lustre-OST0000: new disk, initializing [ 6046.427732] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6046.429865] Lustre: 319565:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6046.467761] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6051.993128] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6053.407678] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6053.415814] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6053.479997] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6063.436723] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6065.884240] Lustre: Mounted lustre-client - version 2.17.57_44_gac22094 [ 6066.714560] Lustre: Unmounted lustre-client [ 6076.388755] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6076.392214] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6076.398032] Lustre: Skipped 1 previous similar message [ 6076.407138] Lustre: Skipped 1 previous similar message [ 6081.513603] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6081.518411] Lustre: Skipped 1 previous similar message [ 6086.631166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6086.645207] Lustre: Skipped 1 previous similar message [ 6089.188027] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6089.325168] Lustre: server umount lustre-OST0000 complete [ 6096.352472] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6096.361306] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6096.370572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6100.524900] Lustre: server umount lustre-MDT0000 complete [ 6104.106167] LustreError: 317589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787999054 with bad export cookie 14578640536134286623 [ 6104.112093] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6104.122350] LustreError: 317589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6104.464711] Lustre: server umount lustre-MDT0001 complete [ 6116.570434] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 6120.057449] Key type lgssc unregistered [ 6123.551708] LNet: 321730:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6123.573396] LNetError: 321730:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6194.278609] LNet: Removed LNI 192.168.202.128@tcp [ 6195.001297] Key type .llcrypt unregistered [ 6195.002878] Key type ._llcrypt unregistered [ 6206.603469] Key type ._llcrypt registered [ 6206.604552] Key type .llcrypt registered [ 6206.687368] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 6217.550315] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 6218.244119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6218.332531] alg: No test for adler32 (adler32-zlib) [ 6219.295568] Lustre: Lustre: Build Version: 2.17.57_44_gac22094 [ 6219.465497] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 6221.119183] Key type lgssc registered [ 6221.856447] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6242.170531] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 06:26:31 (1787999191) [ 6246.501663] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6246.519971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6247.716842] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6247.737305] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6247.792149] Lustre: lustre-MDT0000: new disk, initializing [ 6247.839782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6247.851681] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6251.227760] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6259.913694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6260.034552] Lustre: 324928:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6260.087709] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6260.090872] Lustre: Skipped 1 previous similar message [ 6260.152811] Lustre: lustre-MDT0001: new disk, initializing [ 6260.240702] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6260.274328] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6260.296618] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6264.065199] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6271.092718] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6275.154744] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6280.597288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6280.781594] Lustre: lustre-OST0000: new disk, initializing [ 6280.785565] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6280.790879] Lustre: 326893:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6280.865858] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6285.029490] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6287.396264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6287.401544] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6287.482503] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6292.158884] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6303.201378] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6303.204741] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6303.212261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6307.808479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6307.819131] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6307.838464] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6309.106639] Lustre: server umount lustre-OST0000 complete [ 6312.929500] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6312.931660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6312.933322] Lustre: Skipped 1 previous similar message [ 6312.938883] Lustre: Skipped 2 previous similar messages [ 6318.050030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6318.063215] Lustre: Skipped 1 previous similar message [ 6323.168361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6326.239118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6326.411577] Lustre: server umount lustre-MDT0000 complete [ 6329.645426] LustreError: 324920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787999280 with bad export cookie 3639865095786443641 [ 6329.653118] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6329.666793] LustreError: 324920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6329.913686] Lustre: server umount lustre-MDT0001 complete [ 6336.732264] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 6338.498390] Key type lgssc unregistered [ 6338.706558] LNet: 328803:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6338.715604] LNetError: 328803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6338.728715] LNet: Removed LNI 192.168.202.128@tcp [ 6339.295475] Key type .llcrypt unregistered [ 6339.297642] Key type ._llcrypt unregistered [ 6353.973495] Key type ._llcrypt registered [ 6353.975680] Key type .llcrypt registered [ 6354.067766] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 6355.053685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6355.098608] alg: No test for adler32 (adler32-zlib) [ 6356.084829] Lustre: Lustre: Build Version: 2.17.57_44_gac22094 [ 6356.303476] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 6357.976591] Key type lgssc registered [ 6358.822152] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6364.881760] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6364.897150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6366.177212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6369.174645] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6375.145514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6375.395278] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6375.489123] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6378.425149] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6383.644895] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6387.837949] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6392.282455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6392.574521] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6397.023648] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6402.532885] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6419.121491] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6420.085581] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6423.396457] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.28@tcp (stopping) [ 6426.601164] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6426.604527] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6426.604943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6426.607937] Lustre: Skipped 1 previous similar message [ 6429.355598] Lustre: server umount lustre-MDT0000 complete [ 6431.717945] LustreError: 330057:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6431.731282] LustreError: 330057:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6432.111227] LustreError: 330037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787999382 with bad export cookie 2345951430342561851 [ 6432.118321] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6432.119626] LustreError: 330037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6433.624359] LustreError: 330053:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6433.643133] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.28@tcp (stopping) [ 6433.659900] Lustre: Skipped 3 previous similar messages [ 6436.836829] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6436.839634] LustreError: 330054:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6436.850775] Lustre: Skipped 1 previous similar message [ 6437.863469] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6437.867827] Lustre: Skipped 1 previous similar message [ 6438.449654] Lustre: server umount lustre-MDT0001 complete [ 6447.732677] Lustre: server umount lustre-OST0000 complete [ 6454.252761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6458.524186] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6461.332794] Lustre: 334257:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.128@tcp [ 6463.461567] Lustre: server umount MGS complete [ 6471.490194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6472.043200] LustreError: 334829:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6472.076626] LustreError: 334829:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6472.163846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6476.119734] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6477.281639] LustreError: 334830:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6477.299026] LustreError: 334830:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6484.044158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6484.324701] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6488.318705] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6496.578649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6496.740253] LustreError: 336264:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6496.751334] LustreError: 336264:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6496.870340] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6502.638982] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6509.274490] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6511.138216] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [ 6517.340404] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6519.119683] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6523.880266] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6523.890834] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6523.918345] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6525.364678] Lustre: server umount lustre-OST0000 complete [ 6528.993587] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6529.022131] Lustre: Skipped 1 previous similar message [ 6543.327658] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6543.333974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6543.347889] Lustre: Skipped 8 previous similar messages [ 6543.501077] Lustre: server umount lustre-MDT0000 complete [ 6546.254888] LustreError: 335564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787999497 with bad export cookie 2345951430342562691 [ 6546.255941] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6546.262148] LustreError: 335564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6546.570684] Lustre: server umount lustre-MDT0001 complete [ 6555.437436] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 6557.752458] Key type lgssc unregistered [ 6558.039534] LNet: 338269:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6558.045534] LNetError: 338269:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6559.078035] LNet: Removed LNI 192.168.202.128@tcp [ 6559.936899] Key type .llcrypt unregistered [ 6559.938541] Key type ._llcrypt unregistered [ 6575.782805] Key type ._llcrypt registered [ 6575.784729] Key type .llcrypt registered [ 6575.867124] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 6586.749541] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 6587.465766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6587.494919] alg: No test for adler32 (adler32-zlib) [ 6588.466313] Lustre: Lustre: Build Version: 2.17.57_44_gac22094 [ 6588.627959] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 6590.287209] Key type lgssc registered [ 6591.080645] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6615.280595] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6615.319368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6616.507747] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6616.536479] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6616.587230] Lustre: lustre-MDT0000: new disk, initializing [ 6616.637822] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6616.650909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6620.417788] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6628.493360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6628.569633] Lustre: 341404:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6628.607882] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6628.613853] Lustre: Skipped 1 previous similar message [ 6628.657955] Lustre: lustre-MDT0001: new disk, initializing [ 6628.732367] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6628.759817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6628.772365] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6631.302632] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6637.857224] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6641.496805] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6646.100979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6646.301782] Lustre: lustre-OST0000: new disk, initializing [ 6646.304742] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6646.308696] Lustre: 343369:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6646.378497] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6650.557485] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6651.954700] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6651.958025] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6652.011674] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6656.643553] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6659.771030] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6659.915767] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6662.668499] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6662.820059] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6667.237408] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6667.239693] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6667.263700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6667.264832] LustreError: Skipped 1 previous similar message [ 6667.277977] Lustre: Skipped 1 previous similar message [ 6670.265758] Lustre: server umount lustre-OST0000 complete [ 6674.913458] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6674.924968] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6674.932931] Lustre: Skipped 1 previous similar message [ 6674.935628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6677.474314] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6677.477396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6677.491932] Lustre: Skipped 1 previous similar message [ 6682.592213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6682.599801] Lustre: Skipped 1 previous similar message [ 6687.199171] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6687.355958] Lustre: server umount lustre-MDT0000 complete [ 6689.979652] LustreError: 341396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787999640 with bad export cookie 13515233359843665753 [ 6689.985444] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6689.987779] LustreError: 341396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6690.248990] Lustre: server umount lustre-MDT0001 complete [ 6696.537708] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 06:34:05 (1787999645) [ 6707.289948] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 6713.292297] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 6738.210872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6738.410969] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6738.427244] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6738.478571] Lustre: lustre-MDT0000: new disk, initializing [ 6738.538925] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6738.553174] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6742.049127] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6750.911789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6751.041324] Lustre: 348129:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6751.082108] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6751.087388] Lustre: Skipped 1 previous similar message [ 6751.152765] Lustre: lustre-MDT0001: new disk, initializing [ 6751.260917] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6751.291256] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6751.310973] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6754.728793] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6762.126539] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6765.414698] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6769.297785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6769.454074] Lustre: lustre-OST0000: new disk, initializing [ 6769.457727] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6769.462453] Lustre: 350094:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6769.511955] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6773.086113] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6776.842707] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6776.851553] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6776.910089] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6778.722140] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6792.162069] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6792.163575] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6792.169871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6792.176820] Lustre: Skipped 1 previous similar message [ 6792.186836] Lustre: Skipped 1 previous similar message [ 6798.012782] Lustre: server umount lustre-OST0000 complete [ 6802.400434] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6802.408797] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6802.410609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6802.410615] Lustre: Skipped 2 previous similar messages [ 6802.411789] LustreError: Skipped 1 previous similar message [ 6802.440513] Lustre: Skipped 1 previous similar message [ 6806.287823] Lustre: server umount lustre-MDT0000 complete [ 6807.520263] LustreError: 348140:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6807.527241] LustreError: 348140:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6808.446063] LustreError: 348119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787999759 with bad export cookie 13515233359843666635 [ 6808.446987] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6808.454202] LustreError: 348119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6808.624500] Lustre: server umount lustre-MDT0001 complete [ 6813.738820] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 6815.191988] Key type lgssc unregistered [ 6815.361441] LNet: 352244:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6815.364195] LNetError: 352244:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6815.375466] LNet: Removed LNI 192.168.202.128@tcp [ 6815.719906] Key type .llcrypt unregistered [ 6815.721565] Key type ._llcrypt unregistered [ 6869.703188] Key type ._llcrypt registered [ 6869.704967] Key type .llcrypt registered [ 6869.789805] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 6870.017632] Key type .llcrypt unregistered [ 6870.021132] Key type ._llcrypt unregistered [ 6916.265746] Key type ._llcrypt registered [ 6916.267467] Key type .llcrypt registered [ 6916.341868] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 6916.585585] Key type .llcrypt unregistered [ 6916.590084] Key type ._llcrypt unregistered [ 6933.139391] Key type ._llcrypt registered [ 6933.140987] Key type .llcrypt registered [ 6933.227849] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 6933.428493] Key type .llcrypt unregistered [ 6933.431569] Key type ._llcrypt unregistered [ 6950.751580] Key type ._llcrypt registered [ 6950.753122] Key type .llcrypt registered [ 6950.828043] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 6951.020366] Key type .llcrypt unregistered [ 6951.023924] Key type ._llcrypt unregistered [ 6996.215011] Key type ._llcrypt registered [ 6996.215928] Key type .llcrypt registered [ 6996.281602] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 6996.484652] Key type .llcrypt unregistered [ 6996.486424] Key type ._llcrypt unregistered [ 7013.917883] Key type ._llcrypt registered [ 7013.920670] Key type .llcrypt registered [ 7014.006814] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 7014.226844] Key type .llcrypt unregistered [ 7014.228684] Key type ._llcrypt unregistered [ 7030.956626] Key type ._llcrypt registered [ 7030.960147] Key type .llcrypt registered [ 7031.029526] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 7031.208674] Key type .llcrypt unregistered [ 7031.212055] Key type ._llcrypt unregistered [ 7048.117751] Key type ._llcrypt registered [ 7048.118935] Key type .llcrypt registered [ 7048.178233] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 7048.341353] Key type .llcrypt unregistered [ 7048.343291] Key type ._llcrypt unregistered [ 7066.670197] Key type ._llcrypt registered [ 7066.671784] Key type .llcrypt registered [ 7066.756712] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 7066.951327] Key type .llcrypt unregistered [ 7066.953772] Key type ._llcrypt unregistered [ 7084.368344] Key type ._llcrypt registered [ 7084.370027] Key type .llcrypt registered [ 7084.460547] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 7084.682266] Key type .llcrypt unregistered [ 7084.683328] Key type ._llcrypt unregistered [ 7118.820186] Key type ._llcrypt registered [ 7118.821827] Key type .llcrypt registered [ 7118.884818] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 7119.089437] Key type .llcrypt unregistered [ 7119.091390] Key type ._llcrypt unregistered [ 7138.067969] Key type ._llcrypt registered [ 7138.069245] Key type .llcrypt registered [ 7138.121791] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 7138.291132] Key type .llcrypt unregistered [ 7138.292964] Key type ._llcrypt unregistered [ 7157.280721] Key type ._llcrypt registered [ 7157.282497] Key type .llcrypt registered [ 7157.363385] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 7157.597373] Key type .llcrypt unregistered [ 7157.602584] Key type ._llcrypt unregistered [ 7189.469868] Key type ._llcrypt registered [ 7189.471704] Key type .llcrypt registered [ 7189.559365] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 7198.976205] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 7199.659248] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7199.695687] alg: No test for adler32 (adler32-zlib) [ 7200.595979] Lustre: Lustre: Build Version: 2.17.57_44_gac22094 [ 7200.729401] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 7202.351243] Key type lgssc registered [ 7203.069540] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7224.496642] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7224.525166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7225.697407] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 7225.716187] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 7225.783760] Lustre: tst250fs-MDT0000: new disk, initializing [ 7225.841705] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7225.857654] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7228.042977] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 7235.061122] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7235.165587] Lustre: 372228:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7235.205446] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 7235.208711] Lustre: Skipped 1 previous similar message [ 7235.266142] Lustre: tst250fs-MDT0001: new disk, initializing [ 7235.329299] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7235.352179] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7235.361593] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7237.956511] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 7241.537654] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7245.962938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7246.210966] Lustre: tst250fs-OST0000: new disk, initializing [ 7246.214858] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 7246.221115] Lustre: 373858:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7246.292770] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7250.237950] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 7254.059440] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7254.064178] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7254.128773] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7257.265793] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7257.350868] Lustre: tst250fs-OST0001: new disk, initializing [ 7257.353731] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 7257.357881] Lustre: 374727:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7257.413071] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7260.524262] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 7261.216517] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7261.223513] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7261.270895] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7264.013409] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 7281.633474] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7281.634733] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7281.638601] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 7281.645465] Lustre: Skipped 3 previous similar messages [ 7283.545896] Lustre: server umount tst250fs-MDT0000 complete [ 7285.627263] LustreError: 373857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788000236 with bad export cookie 734849959191695769 [ 7285.632667] LustreError: 373857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7285.633734] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7285.834664] Lustre: server umount tst250fs-MDT0001 complete [ 7297.434066] Lustre: server umount tst250fs-OST0000 complete [ 7310.012418] Lustre: server umount tst250fs-OST0001 complete [ 7319.979578] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 7324.081501] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 7341.552660] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 06:44:50 (1788000290) [ 7342.503792] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 7353.012426] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 7358.465311] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 7377.043582] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7059 sec ========== 06:45:26 (1788000326) [ 7378.278849] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 06:45:27 (1788000327) === [ 7380.494369] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 06:45:29 (1788000329) === [ 7403.124262] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 7404.923370] Key type lgssc unregistered [ 7405.145870] LNet: 383845:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7405.151492] LNetError: 383845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7406.181866] LNet: Removed LNI 192.168.202.128@tcp [ 7406.671348] Key type .llcrypt unregistered [ 7406.672780] Key type ._llcrypt unregistered