[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 474995810 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002297] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004012] kvm-guest: setup PV IPIs [ 0.007623] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008006] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010012] pid_max: default: 32768 minimum: 301 [ 0.012126] LSM: Security Framework initializing [ 0.013050] Yama: becoming mindful. [ 0.014031] SELinux: Initializing. [ 0.015059] *** VALIDATE selinux *** [ 0.023051] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027563] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029056] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031071] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032103] *** VALIDATE tmpfs *** [ 0.034337] *** VALIDATE proc *** [ 0.035214] *** VALIDATE cgroup *** [ 0.036008] *** VALIDATE cgroup2 *** [ 0.037256] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038158] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040035] Spectre V2 : User space: Vulnerable [ 0.041009] Speculative Store Bypass: Vulnerable [ 0.044043] debug: unmapping init [mem 0xffffffffb7c59000-0xffffffffb7c60fff] [ 0.046824] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047652] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048027] ... version: 2 [ 0.049011] ... bit width: 48 [ 0.050010] ... generic registers: 4 [ 0.051012] ... value mask: 0000ffffffffffff [ 0.052012] ... max period: 00007fffffffffff [ 0.053015] ... fixed-purpose events: 3 [ 0.054012] ... event mask: 000000070000000f [ 0.055275] rcu: Hierarchical SRCU implementation. [ 0.057350] smp: Bringing up secondary CPUs ... [ 0.058582] x86: Booting SMP configuration: [ 0.059021] .... node #0, CPUs: #1 #2 #3 [ 0.062479] smp: Brought up 1 node, 4 CPUs [ 0.064011] smpboot: Max logical packages: 1 [ 0.065016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.132721] node 0 deferred pages initialised in 63ms [ 0.137096] devtmpfs: initialized [ 0.138419] x86/mm: Memory block size: 128MB [ 0.142183] gcov: version magic: 0x41383552 [ 0.145052] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.146081] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.148436] pinctrl core: initialized pinctrl subsystem [ 0.151192] [ 0.151846] ************************************************************* [ 0.154012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.156010] ** ** [ 0.159012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.161010] ** ** [ 0.164013] ** This means that this kernel is built to expose internal ** [ 0.166013] ** IOMMU data structures, which may compromise security on ** [ 0.169014] ** your system. ** [ 0.171008] ** ** [ 0.173011] ** If you see this message and you are not debugging the ** [ 0.175014] ** kernel, report this immediately to your vendor! ** [ 0.178018] ** ** [ 0.180011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.183013] ************************************************************* [ 0.186917] NET: Registered protocol family 16 [ 0.188610] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.192146] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.195081] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.199195] cpuidle: using governor menu [ 0.201908] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.205655] PCI: Using configuration type 1 for base access [ 0.207140] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.217156] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.220033] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.223037] cryptd: max_cpu_qlen set to 1000 [ 0.225297] ACPI: Added _OSI(Module Device) [ 0.227012] ACPI: Added _OSI(Processor Device) [ 0.229005] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.230011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.231000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.238525] ACPI: Interpreter enabled [ 0.240065] ACPI: PM: (supports S0 S3 S4 S5) [ 0.242014] ACPI: Using IOAPIC for interrupt routing [ 0.244111] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.248392] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.258884] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.261038] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.264019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.268138] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.274595] acpiphp: Slot [2] registered [ 0.276145] acpiphp: Slot [5] registered [ 0.278138] acpiphp: Slot [6] registered [ 0.280153] acpiphp: Slot [7] registered [ 0.281168] acpiphp: Slot [8] registered [ 0.283194] acpiphp: Slot [9] registered [ 0.285188] acpiphp: Slot [10] registered [ 0.286156] acpiphp: Slot [3] registered [ 0.288125] acpiphp: Slot [4] registered [ 0.290140] acpiphp: Slot [11] registered [ 0.292239] acpiphp: Slot [12] registered [ 0.293112] acpiphp: Slot [13] registered [ 0.295141] acpiphp: Slot [14] registered [ 0.297204] acpiphp: Slot [15] registered [ 0.298130] acpiphp: Slot [16] registered [ 0.300144] acpiphp: Slot [17] registered [ 0.302108] acpiphp: Slot [18] registered [ 0.303133] acpiphp: Slot [19] registered [ 0.305097] acpiphp: Slot [20] registered [ 0.306102] acpiphp: Slot [21] registered [ 0.308126] acpiphp: Slot [22] registered [ 0.310106] acpiphp: Slot [23] registered [ 0.311189] acpiphp: Slot [24] registered [ 0.313104] acpiphp: Slot [25] registered [ 0.315098] acpiphp: Slot [26] registered [ 0.316097] acpiphp: Slot [27] registered [ 0.318118] acpiphp: Slot [28] registered [ 0.319122] acpiphp: Slot [29] registered [ 0.321103] acpiphp: Slot [30] registered [ 0.323115] acpiphp: Slot [31] registered [ 0.325064] PCI host bridge to bus 0000:00 [ 0.326019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.329021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.332027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.334019] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.337037] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.340040] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.343226] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.347218] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.351435] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.361813] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.367063] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.370036] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.373025] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.376028] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.380030] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.383054] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.387064] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.390906] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.397030] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.409022] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.415021] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.422959] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.430016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.436017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.451021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.461357] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.469018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.476016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.496018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.509251] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.517022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.528022] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.553023] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.572831] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.583023] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.593022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.612024] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.625497] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.634017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.640016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.654015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.666051] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.681025] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.690020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.715025] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.745329] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.749376] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.751328] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.754350] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.756219] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.762030] iommu: Default domain type: Passthrough [ 0.763349] SCSI subsystem initialized [ 0.765121] ACPI: bus type USB registered [ 0.767169] usbcore: registered new interface driver usbfs [ 0.769075] usbcore: registered new interface driver hub [ 0.771100] usbcore: registered new device driver usb [ 0.773172] pps_core: LinuxPPS API ver. 1 registered [ 0.775012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.778054] PTP clock support registered [ 0.780198] EDAC MC: Ver: 3.0.0 [ 0.782487] PCI: Using ACPI for IRQ routing [ 0.785064] NetLabel: Initializing [ 0.787019] NetLabel: domain hash size = 128 [ 0.789017] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.791170] NetLabel: unlabeled traffic allowed by default [ 0.794486] vgaarb: loaded [ 0.796333] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.798023] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.810674] clocksource: Switched to clocksource kvm-clock [ 0.910975] VFS: Disk quotas dquot_6.6.0 [ 0.912880] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.915408] *** VALIDATE ramfs *** [ 0.916743] *** VALIDATE hugetlbfs *** [ 0.918486] pnp: PnP ACPI init [ 0.920685] pnp: PnP ACPI: found 6 devices [ 0.938742] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.942665] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.945037] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.947218] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.949887] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.952526] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.954694] NET: Registered protocol family 2 [ 0.957259] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.962068] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.966077] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.972465] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.976740] TCP: Hash tables configured (established 65536 bind 65536) [ 0.980359] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.984173] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.987747] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.991678] NET: Registered protocol family 1 [ 0.995917] RPC: Registered named UNIX socket transport module. [ 0.998364] RPC: Registered udp transport module. [ 1.000606] RPC: Registered tcp transport module. [ 1.002675] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.005680] NET: Registered protocol family 44 [ 1.007593] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.010101] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.012782] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.015623] PCI: CLS 0 bytes, default 64 [ 1.017613] Unpacking initramfs... [ 2.456808] debug: unmapping init [mem 0xffff8cb93cc54000-0xffff8cb93ffbffff] [ 2.461637] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.464560] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.468130] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.977293] Initialise system trusted keyrings [ 2.979107] Key type blacklist registered [ 2.981462] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.990054] zbud: loaded [ 2.993290] *** VALIDATE nfs *** [ 2.994743] *** VALIDATE nfs4 *** [ 2.996404] pstore: using deflate compression [ 2.999866] Platform Keyring initialized [ 3.120906] NET: Registered protocol family 38 [ 3.124226] Key type asymmetric registered [ 3.126096] Asymmetric key parser 'x509' registered [ 3.128710] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.132358] io scheduler mq-deadline registered [ 3.134105] io scheduler kyber registered [ 3.136046] io scheduler bfq registered [ 3.138147] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.142261] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.145939] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.149439] ACPI: Power Button [PWRF] [ 3.157283] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.164493] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.181864] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.195567] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.217469] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.247226] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.276491] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.283139] Non-volatile memory driver v1.3 [ 3.285101] Linux agpgart interface v0.103 [ 3.325390] virtio_blk virtio1: [vda] 139280 512-byte logical blocks (71.3 MB/68.0 MiB) [ 3.330660] vda: detected capacity change from 0 to 71311360 [ 3.349380] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.352883] vdb: detected capacity change from 0 to 1073741824 [ 3.371560] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.374338] vdc: detected capacity change from 0 to 2621440000 [ 3.392143] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.395588] vdd: detected capacity change from 0 to 2621440000 [ 3.410365] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.412806] vde: detected capacity change from 0 to 4294967296 [ 3.432361] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.437434] vdf: detected capacity change from 0 to 4294967296 [ 3.443368] libphy: Fixed MDIO Bus: probed [ 3.449768] usbcore: registered new interface driver usbserial_generic [ 3.452878] usbserial: USB Serial support registered for generic [ 3.455542] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.461851] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.464161] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.467177] mousedev: PS/2 mouse device common for all mice [ 3.470921] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.471855] rtc_cmos 00:05: RTC can wake from S4 [ 3.480422] rtc_cmos 00:05: registered as rtc0 [ 3.482994] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.486097] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.487361] intel_pstate: CPU model not supported [ 3.494037] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.496157] hid: raw HID events driver (C) Jiri Kosina [ 3.500594] usbcore: registered new interface driver usbhid [ 3.502994] usbhid: USB HID core driver [ 3.504979] drop_monitor: Initializing network drop monitor service [ 3.507991] Initializing XFRM netlink socket [ 3.510428] NET: Registered protocol family 10 [ 3.515093] Segment Routing with IPv6 [ 3.516830] NET: Registered protocol family 17 [ 3.519515] mpls_gso: MPLS GSO support [ 3.526130] RAS: Correctable Errors collector initialized. [ 3.528728] AVX version of gcm_enc/dec engaged. [ 3.530801] AES CTR mode by8 optimization enabled [ 3.619600] sched_clock: Marking stable (3619558984, 0)->(4517586829, -898027845) [ 3.623971] registered taskstats version 1 [ 3.627662] Loading compiled-in X.509 certificates [ 3.630080] zswap: loaded using pool lzo/zbud [ 3.656849] Key type big_key registered [ 3.671551] Key type encrypted registered [ 3.673506] ima: No TPM chip found, activating TPM-bypass! [ 3.675883] ima: Allocated hash algorithm: sha1 [ 3.677825] ima: No architecture policies found [ 3.679963] evm: Initialising EVM extended attributes: [ 3.682097] evm: security.selinux [ 3.683606] evm: security.ima [ 3.684882] evm: security.capability [ 3.686416] evm: HMAC attrs: 0x1 [ 3.690587] rtc_cmos 00:05: setting system clock to 2026-06-16 04:14:13 UTC (1781583253) [ 3.698336] debug: unmapping init [mem 0xffffffffb8c03000-0xffffffffb8dfffff] [ 3.702230] debug: unmapping init [mem 0xffffffffb7982000-0xffffffffb7c58fff] [ 3.712277] Write protecting the kernel read-only data: 28672k [ 3.716069] debug: unmapping init [mem 0xffffffffb6003000-0xffffffffb61fffff] [ 3.718976] debug: unmapping init [mem 0xffffffffb6914000-0xffffffffb69fffff] [ 3.758991] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.768417] systemd[1]: Detected virtualization kvm. [ 3.770701] systemd[1]: Detected architecture x86-64. [ 3.772964] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.801637] systemd[1]: No hostname configured. [ 3.803542] systemd[1]: Set hostname to . [ 3.805894] random: systemd: uninitialized urandom read (16 bytes read) [ 3.809148] systemd[1]: Initializing machine ID from random generator. [ 3.944298] random: systemd: uninitialized urandom read (16 bytes read) [ 3.947617] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.952512] random: systemd: uninitialized urandom read (16 bytes read) [ 3.955750] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.963606] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local File Systems. [ OK ] Reached target Slices. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Journal Service... Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.613598] device-mapper: uevent: version 1.0.3 [ 4.616505] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.363417] virtio_net virtio0 ens2: renamed from eth0 [ 5.365140] random: fast init done Starting dracut initqueue hook... [ 5.493570] scsi host0: ata_piix [ 5.523075] scsi host1: ata_piix [ 5.525101] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.527398] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.804384] dracut-initqueue[591]: RTNETLINK answers: File exists [ 10.194496] random: crng init done [ 10.196331] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.722193] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.054385] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.373527] SELinux: Disabled at runtime. [ 12.439604] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.446529] systemd[1]: Detected virtualization kvm. [ 12.448027] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.009556] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.013273] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.018560] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.024363] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.028589] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.037063] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.058802] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting POSIX Message Queue File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-getty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... [ OK ] Created slic[ 13.154472] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS e system-sshd\x2dkeygen.slice. Starting Remount Root and Kernel File Systems... Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ 13.523201] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /mnt. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.898718] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.921994] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.212164] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.233831] EDAC sbridge: Ver: 1.1.2 [ 16.005363] Key type dns_resolver registered [ 16.321994] NFS: Registering the id_resolver key type [ 16.324459] Key type id_resolver registered [ 16.326342] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. [ OK ] Started dnf makecache --timer. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started OpenSSH server daemon. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg649-server login: [ 101.938132] hrtimer: interrupt took 3078692 ns [ 118.036983] libcfs: loading out-of-tree module taints kernel. [ 118.189100] Key type ._llcrypt registered [ 118.192687] Key type .llcrypt registered [ 118.407946] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_hostid [ 144.178253] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 146.724464] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 146.755586] alg: No test for adler32 (adler32-zlib) [ 148.773757] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 150.049845] LNet: Added LNI 192.168.206.149@tcp [8/256/0/180] [ 151.819098] Key type lgssc registered [ 154.111751] Lustre: Echo OBD driver; http://www.lustre.org/ [ 177.859615] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 231.401291] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 249.396286] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 249.458849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 250.818945] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 250.887672] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 251.004627] Lustre: lustre-MDT0000: new disk, initializing [ 251.136136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 251.161928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 257.559847] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 276.462357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 276.586565] Lustre: 6527:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 276.623925] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 276.629357] Lustre: Skipped 1 previous similar message [ 276.738555] Lustre: lustre-MDT0001: new disk, initializing [ 276.823526] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 276.847607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 276.865389] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 283.958911] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 290.558916] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 305.990460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 306.526048] Lustre: lustre-OST0000: new disk, initializing [ 306.533966] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 306.542157] Lustre: 8469:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 306.724545] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 312.928627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 312.959368] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 313.130341] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 315.416430] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 335.460149] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 335.677181] Lustre: lustre-OST0001: new disk, initializing [ 335.688932] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 335.702353] Lustre: 9543:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 335.839336] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 342.594597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 342.604708] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 342.730432] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 343.947507] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 360.442099] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 369.138125] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 377.254690] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing check_logdir /tmp/testlogs/ [ 384.116839] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing yml_node [ 390.083907] Lustre: DEBUG MARKER: Client: 2.17.54.40 [ 393.323901] Lustre: DEBUG MARKER: MDS: 2.17.54.40 [ 396.723725] Lustre: DEBUG MARKER: OSS: 2.17.54.40 [ 398.969865] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Jun 16 00:20:46 EDT 2026 [ 420.142352] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 422.399883] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 440.318474] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 440.327639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 440.341253] Lustre: Skipped 2 previous similar messages [ 440.357483] Lustre: Skipped 3 previous similar messages [ 445.417768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 445.425752] Lustre: Skipped 3 previous similar messages [ 446.322683] Lustre: server umount lustre-MDT0000 complete [ 451.141099] LustreError: 6518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781583700 with bad export cookie 18396062332628191227 [ 451.141442] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 451.155474] LustreError: 6518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 455.666598] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 455.666845] LustreError: 12180:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 455.677515] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 455.678126] Lustre: Skipped 2 previous similar messages [ 455.736637] LustreError: 12180:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 456.678567] LustreError: 6535:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 456.705652] LustreError: 6535:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 458.203458] Lustre: server umount lustre-MDT0001 complete [ 463.839806] Lustre: server umount lustre-OST0000 complete [ 469.249573] Lustre: server umount lustre-OST0001 complete [ 478.045921] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_hostid [ 490.739444] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 537.716568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 538.194739] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 538.262638] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 538.441617] Lustre: lustre-MDT0000: new disk, initializing [ 538.580445] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 538.633954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 544.790113] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 557.726732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 557.879858] Lustre: 15101:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 557.947349] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 557.953769] Lustre: Skipped 1 previous similar message [ 558.065331] Lustre: lustre-MDT0001: new disk, initializing [ 558.181364] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 558.235471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 558.251417] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 563.353656] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 575.753695] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 583.195963] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 592.689935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 593.202412] Lustre: lustre-OST0000: new disk, initializing [ 593.214813] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 593.227959] Lustre: 17072:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 593.389760] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 594.872816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 594.894470] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 595.063387] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 601.079891] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 612.411652] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 618.354681] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 618.646385] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 625.587679] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 625.862770] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 630.784772] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 630.788932] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 630.797872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 630.797881] Lustre: Skipped 3 previous similar messages [ 630.830136] Lustre: Skipped 1 previous similar message [ 630.865591] LustreError: Skipped 1 previous similar message [ 635.662691] Lustre: server umount lustre-OST0000 complete [ 641.000909] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 641.014158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 641.021345] Lustre: Skipped 1 previous similar message [ 641.054388] Lustre: Skipped 2 previous similar messages [ 650.215759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 650.227243] Lustre: Skipped 3 previous similar messages [ 655.328996] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 655.733304] Lustre: server umount lustre-MDT0000 complete [ 660.449925] LustreError: 15113:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 660.499882] LustreError: 15113:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 660.720416] LustreError: 15094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781583910 with bad export cookie 18396062332628192760 [ 660.722175] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 660.737782] LustreError: 15094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 661.482342] Lustre: server umount lustre-MDT0001 complete [ 671.022903] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 00:25:17 (1781583917) [ 696.098774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 696.803059] LustreError: 19949:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 696.943948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 702.437149] LustreError: 19950:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 703.565175] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 711.671404] LustreError: 19950:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 711.698472] LustreError: 19950:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 715.841773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 722.401080] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 732.421746] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 739.269878] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 741.789345] Lustre: Failing over lustre-MDT0000 [ 741.804369] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 741.807335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 741.815669] Lustre: Skipped 1 previous similar message [ 741.831508] Lustre: Skipped 3 previous similar messages [ 742.111262] Lustre: server umount lustre-MDT0000 complete [ 742.889954] LustreError: 19945:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 742.913652] LustreError: 19945:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 758.240118] Lustre: 3656:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781583992/real 1781583992] req@ffff8cb9bcc71880 x1868125594369536/t0(0) o400->MGC192.168.206.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781584008 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 758.282865] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 762.239755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 769.519292] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xff4bf1399a3f537e [ 769.541210] Lustre: MGC192.168.206.149@tcp: Connection restored to 0@lo (at 0@lo) [ 769.893545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 769.899517] Lustre: Skipped 1 previous similar message [ 769.962053] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 775.141613] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 775.154305] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 775.202211] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 775.451430] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 786.115476] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 790.498374] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 790.509310] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 790.530302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 796.696730] Lustre: server umount lustre-MDT0000 complete [ 797.671169] LustreError: 19946:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 797.711563] LustreError: 19946:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 17 previous similar messages [ 801.504937] LustreError: 19930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781584051 with bad export cookie 18396062332628194174 [ 801.512940] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 801.518534] LustreError: 19930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 801.865621] Lustre: server umount lustre-MDT0001 complete [ 812.126805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 818.549367] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 830.194301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 836.866140] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 846.816156] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 853.513748] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 855.909398] Lustre: Failing over lustre-MDT0000 [ 856.416441] Lustre: server umount lustre-MDT0000 complete [ 856.552083] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 856.572380] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 856.603807] Lustre: Skipped 1 previous similar message [ 865.773821] LustreError: 23291:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 865.797127] LustreError: 23291:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 877.030006] Lustre: 3658:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781584110/real 1781584110] req@ffff8cb9bccd7800 x1868125594401280/t0(0) o400->MGC192.168.206.149@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781584126 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 877.073550] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 878.568422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 886.240655] LustreError: 3654:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8cb988f36300 x1868125594403840/t0(0) o250->MGC192.168.206.149@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 886.811750] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 891.884086] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 891.898315] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 891.910685] Lustre: Skipped 1 previous similar message [ 891.951291] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 892.233843] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 904.300969] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 908.885558] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 908.901151] Lustre: Skipped 2 previous similar messages [ 908.902986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 908.914945] Lustre: Skipped 3 previous similar messages [ 915.469629] Lustre: server umount lustre-MDT0000 complete [ 919.941716] LustreError: 23270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781584169 with bad export cookie 18396062332628195112 [ 919.963318] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 920.419838] Lustre: server umount lustre-MDT0001 complete [ 932.595994] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 00:29:39 (1781584179) [ 950.459934] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_hostid [ 960.773796] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 1009.368762] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 1023.394797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1023.804544] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1023.856584] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1023.984782] Lustre: lustre-MDT0000: new disk, initializing [ 1024.136813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1024.153715] Lustre: Skipped 3 previous similar messages [ 1024.188909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1029.120495] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 1041.311469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1041.514389] Lustre: 29672:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1041.986363] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1047.143727] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 1057.384993] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1063.634810] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1071.258668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1071.640870] Lustre: 31641:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1073.036719] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1073.130223] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 1073.239129] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1073.278544] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 1079.969456] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 1091.916135] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1098.733701] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 1124.328187] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1124.330330] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1124.339883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1124.339893] Lustre: Skipped 2 previous similar messages [ 1124.345043] LustreError: Skipped 1 previous similar message [ 1124.395784] Lustre: Skipped 1 previous similar message [ 1127.325912] Lustre: server umount lustre-OST0000 complete [ 1134.050140] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1138.547248] Lustre: server umount lustre-MDT0000 complete [ 1139.682278] LustreError: 29685:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1139.706162] LustreError: 29685:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 1143.416227] LustreError: 29666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781584393 with bad export cookie 18396062332628195518 [ 1143.423800] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1143.429402] LustreError: 29666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 1144.153962] Lustre: server umount lustre-MDT0001 complete [ 1158.699153] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [ 1162.537146] Key type lgssc unregistered [ 1162.863142] LNet: 33623:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1162.873506] LNetError: 33623:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1162.904123] LNet: Removed LNI 192.168.206.149@tcp [ 1163.923338] Key type .llcrypt unregistered [ 1163.928348] Key type ._llcrypt unregistered [ 1192.398915] Key type ._llcrypt registered [ 1192.401437] Key type .llcrypt registered [ 1192.524149] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 1193.647559] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1193.804272] alg: No test for adler32 (adler32-zlib) [ 1194.842152] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 1195.033674] LNet: Added LNI 192.168.206.149@tcp [8/256/0/180] [ 1196.728198] Key type lgssc registered [ 1197.852260] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1206.617821] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1206.636390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1208.321608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1213.615304] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 1218.534569] LustreError: 35053:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1218.550764] LustreError: 35053:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1222.701150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1223.062471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1229.486664] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 1237.945551] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1245.229786] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1255.070518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1255.731154] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1263.881449] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 1268.092442] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1272.914969] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1278.395272] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1297.047433] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 00:35:44 (1781584544) [ 1299.688739] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1302.430285] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1311.676402] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 00:35:59 (1781584559) [ 1323.901611] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 00:36:11 (1781584571) [ 1325.307747] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1335.252996] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1335.257929] Lustre: Skipped 2 previous similar messages [ 1337.462362] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 00:36:25 (1781584585) [ 1340.067821] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1340.074986] Lustre: Skipped 2 previous similar messages [ 1348.164768] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 1348.175275] Lustre: Skipped 1 previous similar message [ 1365.864207] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1365.876302] Lustre: Skipped 5 previous similar messages [ 1370.731878] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 00:36:56 (1781584616) [ 1375.332859] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1380.897049] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1392.119668] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 00:37:19 (1781584639) [ 1399.964057] Lustre: Disabling parameter general.jobid_name=TESTNAME1 in log params [ 1399.973701] Lustre: Skipped 6 previous similar messages [ 1409.541719] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 00:37:36 (1781584656) [ 1426.913867] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1426.915531] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1426.930661] LustreError: Skipped 1 previous similar message [ 1426.931326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1426.943667] Lustre: Skipped 1 previous similar message [ 1426.962227] Lustre: Skipped 1 previous similar message [ 1432.045103] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1432.057270] Lustre: Skipped 1 previous similar message [ 1437.164497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1437.176319] Lustre: Skipped 1 previous similar message [ 1440.224170] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1440.438422] Lustre: server umount lustre-OST0000 complete [ 1447.394771] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1447.405816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1447.409526] Lustre: Skipped 1 previous similar message [ 1447.435752] Lustre: Skipped 1 previous similar message [ 1451.244509] Lustre: server umount lustre-MDT0000 complete [ 1452.514746] LustreError: 36786:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1452.536449] LustreError: 36786:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1455.692472] LustreError: 38638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781584705 with bad export cookie 4201773358297200688 [ 1455.696370] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1455.718717] LustreError: 38638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1456.331543] Lustre: server umount lustre-MDT0001 complete [ 1468.458843] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [ 1471.758417] Key type lgssc unregistered [ 1472.146577] LNet: 42924:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1472.150898] LNetError: 42924:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1472.174130] LNet: Removed LNI 192.168.206.149@tcp [ 1473.320809] Key type .llcrypt unregistered [ 1473.322865] Key type ._llcrypt unregistered [ 1496.807318] Key type ._llcrypt registered [ 1496.809705] Key type .llcrypt registered [ 1496.935103] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_hostid [ 1516.531696] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 1518.056808] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1518.092269] alg: No test for adler32 (adler32-zlib) [ 1519.152460] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 1519.437847] LNet: Added LNI 192.168.206.149@tcp [8/256/0/180] [ 1521.136223] Key type lgssc registered [ 1522.469306] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1565.054530] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1565.097102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1566.602216] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1566.628225] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1566.729399] Lustre: lustre-MDT0000: new disk, initializing [ 1566.803468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1566.823208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1572.463915] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 1584.471591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1584.656909] Lustre: 46089:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1584.775506] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1584.780071] Lustre: Skipped 1 previous similar message [ 1584.871623] Lustre: lustre-MDT0001: new disk, initializing [ 1585.006458] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1585.051326] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1585.063264] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1590.771845] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 1602.106990] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1608.895197] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1617.316482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1617.866511] Lustre: lustre-OST0000: new disk, initializing [ 1617.879847] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1617.897974] Lustre: 48058:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1618.057355] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1625.683653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1625.703394] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1625.914449] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1626.552802] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 1636.637965] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1642.025280] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1642.304174] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1647.712303] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1647.925893] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1651.690431] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1651.704770] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1651.717872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1651.721068] Lustre: Skipped 1 previous similar message [ 1651.741372] Lustre: Skipped 1 previous similar message [ 1656.715738] Lustre: server umount lustre-OST0000 complete [ 1661.922399] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1661.950397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1661.963133] Lustre: Skipped 1 previous similar message [ 1667.044831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1667.054454] Lustre: Skipped 1 previous similar message [ 1667.610363] Lustre: server umount lustre-MDT0000 complete [ 1671.735803] LustreError: 46081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781584921 with bad export cookie 15226223218832472072 [ 1671.739545] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1671.763693] LustreError: 46081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1672.195309] Lustre: server umount lustre-MDT0001 complete [ 1684.162273] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 00:42:11 (1781584931) [ 1691.971615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1692.662941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1697.770151] LustreError: 50519:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1697.800314] LustreError: 50519:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1697.978722] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 1702.886724] LustreError: 50520:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1708.013905] LustreError: 50519:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1708.088262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1708.521508] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1713.916968] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 1722.109147] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1728.130903] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1736.376151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1736.872023] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1744.764594] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 1753.736926] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1758.324204] LustreError: 53050:0:(llog.c:444:llog_init_handle()) llog type is not specified! [ 1759.527905] Lustre: Modifying parameter general.timeout=1 in log params [ 1760.701157] Lustre: Modifying parameter general.timeout=2 in log params [ 1762.769728] Lustre: Modifying parameter general.timeout=4 in log params [ 1762.775264] Lustre: Skipped 1 previous similar message [ 1767.673987] Lustre: Modifying parameter general.timeout=9 in log params [ 1767.687797] Lustre: Skipped 4 previous similar messages [ 1775.693643] Lustre: Modifying parameter general.timeout=17 in log params [ 1775.697853] Lustre: Skipped 7 previous similar messages [ 1792.595678] Lustre: Modifying parameter general.timeout=33 in log params [ 1792.603276] Lustre: Skipped 15 previous similar messages [ 1824.789597] Lustre: Modifying parameter general.timeout=64 in log params [ 1824.794788] Lustre: Skipped 30 previous similar messages [ 1889.473743] Lustre: Modifying parameter general.timeout=120 in log params [ 1889.485817] Lustre: Skipped 55 previous similar messages [ 1911.607488] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 00:45:59 (1781585159) [ 1914.884559] Lustre: 60017:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1916.946090] Lustre: 60115:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1919.067351] Lustre: 60214:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1924.569438] Lustre: 60522:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1924.583106] Lustre: 60522:0:(mgs_llog.c:1447:mgs_modify_param()) Skipped 2 previous similar messages [ 1954.223775] Lustre: 61886:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1954.261950] LustreError: 61886:0:(mgs_handler.c:1202:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1958.948472] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 00:46:46 (1781585206) [ 1974.753534] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1974.763623] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1974.767423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1974.777353] LustreError: Skipped 1 previous similar message [ 1974.810771] Lustre: Skipped 3 previous similar messages [ 1979.427199] Lustre: server umount lustre-MDT0000 complete [ 1979.878073] LustreError: 50515:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1979.899142] LustreError: 50515:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1983.019348] LustreError: 56970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781585232 with bad export cookie 15226223218832472954 [ 1983.020356] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1983.032331] LustreError: 56970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1983.330588] Lustre: server umount lustre-MDT0001 complete [ 1996.610645] Lustre: server umount lustre-OST0000 complete [ 2025.574752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2025.867860] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2025.916446] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2025.935357] Lustre: Skipped 32 previous similar messages [ 2026.127893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2030.882042] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 2039.878217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2039.960771] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2039.997453] Lustre: 64418:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2045.365208] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 2053.516115] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2059.217740] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2066.944670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2067.158185] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2067.370574] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2067.375243] Lustre: Skipped 1 previous similar message [ 2074.294381] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 2082.273918] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2084.332929] Lustre: 66987:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 2095.073443] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2095.087632] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2095.120025] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2095.125994] Lustre: Skipped 2 previous similar messages [ 2097.563507] Lustre: server umount lustre-OST0000 complete [ 2105.826844] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2105.833243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2105.836273] Lustre: Skipped 2 previous similar messages [ 2105.841904] Lustre: Skipped 1 previous similar message [ 2108.547446] Lustre: server umount lustre-MDT0000 complete [ 2110.948101] LustreError: 64430:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2110.969606] LustreError: 64430:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2112.475516] LustreError: 64409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781585362 with bad export cookie 15226223218832476188 [ 2112.483178] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2112.487984] LustreError: 64409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2113.017888] Lustre: server umount lustre-MDT0001 complete [ 2126.020959] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [ 2129.936722] Key type lgssc unregistered [ 2130.325539] LNet: 68200:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2130.333545] LNetError: 68200:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2130.348479] LNet: Removed LNI 192.168.206.149@tcp [ 2131.488851] Key type .llcrypt unregistered [ 2131.496748] Key type ._llcrypt unregistered [ 2162.451253] Key type ._llcrypt registered [ 2162.460171] Key type .llcrypt registered [ 2162.583411] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 2164.262540] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2164.319229] alg: No test for adler32 (adler32-zlib) [ 2165.370571] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 2165.727067] LNet: Added LNI 192.168.206.149@tcp [8/256/0/180] [ 2167.440434] Key type lgssc registered [ 2168.997209] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2177.885542] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2177.906482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2179.331124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2183.676946] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 2189.796478] LustreError: 69618:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2189.805581] LustreError: 69618:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2192.981182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2193.692760] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2197.779318] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 2204.481973] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2210.050225] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2216.415341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2217.046144] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2223.765815] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 2232.096371] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2241.291540] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2254.822195] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2254.839070] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2254.872432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2255.339773] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2255.354649] Lustre: Skipped 1 previous similar message [ 2258.675615] Lustre: server umount lustre-MDT0000 complete [ 2260.456220] LustreError: 69619:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2260.477263] LustreError: 69619:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2262.704961] LustreError: 69600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781585512 with bad export cookie 15639694241853288849 [ 2262.713404] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2262.718431] LustreError: 69600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2263.144054] Lustre: server umount lustre-MDT0001 complete [ 2278.072490] Lustre: server umount lustre-OST0000 complete [ 2307.608821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2307.944422] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2307.992974] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2308.318022] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2313.079631] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 2322.224810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2322.419860] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2322.444876] Lustre: 74491:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2322.926230] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2327.567594] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 2335.460039] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2341.274694] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2348.041679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2348.237041] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2348.553237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2355.325841] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 2362.640540] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2364.354582] Lustre: 77067:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2366.039626] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2375.654717] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2375.663168] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2375.709643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2375.722493] Lustre: Skipped 3 previous similar messages [ 2377.366567] Lustre: server umount lustre-OST0000 complete [ 2381.281938] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2381.284381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2381.297020] Lustre: Skipped 2 previous similar messages [ 2381.303120] Lustre: Skipped 2 previous similar messages [ 2386.406019] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2386.413744] Lustre: Skipped 1 previous similar message [ 2387.262947] Lustre: server umount lustre-MDT0000 complete [ 2391.434989] LustreError: 74483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781585641 with bad export cookie 15639694241853289696 [ 2391.446595] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2391.453362] LustreError: 74483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2391.969059] Lustre: server umount lustre-MDT0001 complete [ 2404.330183] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [ 2407.442094] Key type lgssc unregistered [ 2407.917600] LNet: 78284:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2407.933375] LNetError: 78284:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2409.002539] LNet: Removed LNI 192.168.206.149@tcp [ 2409.849516] Key type .llcrypt unregistered [ 2409.852573] Key type ._llcrypt unregistered [ 2435.878126] Key type ._llcrypt registered [ 2435.880817] Key type .llcrypt registered [ 2435.981447] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 2437.023028] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2437.050605] alg: No test for adler32 (adler32-zlib) [ 2438.200817] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 2438.437362] LNet: Added LNI 192.168.206.149@tcp [8/256/0/180] [ 2440.129056] Key type lgssc registered [ 2441.403181] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2450.386974] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2450.410555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2452.015414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2456.377659] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 2462.180548] LustreError: 79698:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2462.196156] LustreError: 79698:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2465.575401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2466.144962] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2471.169510] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 2479.624544] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2485.884454] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2493.886691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2494.656563] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2501.547417] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 2509.729230] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2516.231365] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2517.101451] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2519.015084] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2519.029757] Lustre: Skipped 1 previous similar message [ 2521.189220] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2521.198552] Lustre: Skipped 1 previous similar message [ 2525.202739] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2525.212226] Lustre: Skipped 3 previous similar messages [ 2533.386515] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2533.399933] Lustre: Skipped 7 previous similar messages [ 2549.639833] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2549.652730] Lustre: Skipped 15 previous similar messages [ 2581.667180] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2581.671174] Lustre: Skipped 30 previous similar messages [ 2627.396726] Lustre: 87806:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2630.418692] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 00:57:57 (1781585877) [ 2632.193873] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2634.296384] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 00:58:01 (1781585881) [ 2639.344536] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2639.368668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2639.372457] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2644.451675] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2644.462648] Lustre: Skipped 2 previous similar messages [ 2644.854894] Lustre: server umount lustre-OST0000 complete [ 2649.581588] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2649.589191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2649.591894] Lustre: Skipped 2 previous similar messages [ 2649.613880] Lustre: Skipped 1 previous similar message [ 2654.690089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2654.701051] Lustre: Skipped 1 previous similar message [ 2654.884580] Lustre: server umount lustre-MDT0000 complete [ 2658.711668] LustreError: 80438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781585908 with bad export cookie 834320066896072234 [ 2658.713933] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2658.725658] LustreError: 80438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2659.149151] Lustre: server umount lustre-MDT0001 complete [ 2670.983481] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [ 2674.040315] Key type lgssc unregistered [ 2674.388281] LNet: 89058:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2674.399912] LNetError: 89058:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2674.431741] LNet: Removed LNI 192.168.206.149@tcp [ 2675.526503] Key type .llcrypt unregistered [ 2675.528915] Key type ._llcrypt unregistered [ 2681.711750] Key type ._llcrypt registered [ 2681.717454] Key type .llcrypt registered [ 2681.840923] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2687.268535] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules [ 2688.946982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2689.099296] alg: No test for adler32 (adler32-zlib) [ 2690.230770] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 2690.529149] LNet: Added LNI 192.168.206.149@tcp [8/256/0/180] [ 2692.320276] Key type lgssc registered [ 2694.077507] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2695.701350] LustreError: 90311:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2715.227352] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 2719.194364] LustreError: 90311:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2725.939783] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2725.958463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2727.548581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2732.367945] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 2738.154294] LustreError: 91342:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2738.177327] LustreError: 91342:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2742.291382] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 00:59:50 (1781585990) [ 2743.270665] LustreError: 91361:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2748.243660] LustreError: 91375:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2748.253449] LustreError: 91375:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2748.260607] LustreError: 91375:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 2748.788941] Lustre: server umount lustre-MDT0000 complete [ 2764.431826] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [ 2768.145510] Key type lgssc unregistered [ 2768.561917] LNet: 92868:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2768.569931] LNetError: 92868:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2768.591193] LNet: Removed LNI 192.168.206.149@tcp [ 2769.653938] Key type .llcrypt unregistered [ 2769.655943] Key type ._llcrypt unregistered [ 2789.833640] Key type ._llcrypt registered [ 2789.836786] Key type .llcrypt registered [ 2789.962956] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 2791.085727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2791.198227] alg: No test for adler32 (adler32-zlib) [ 2792.336636] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 2792.596865] LNet: Added LNI 192.168.206.149@tcp [8/256/0/180] [ 2794.288179] Key type lgssc registered [ 2795.596697] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2804.490826] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2804.533824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2806.039465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2811.805765] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 2815.467072] LustreError: 94089:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2815.504174] LustreError: 94089:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2820.583800] LustreError: 94090:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2822.834100] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2823.652836] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2829.111607] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 2838.466702] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2845.309806] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2855.537975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2856.412342] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2864.522917] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 2874.564249] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2929.188916] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 01:02:56 (1781586176) [ 2931.028288] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2933.266941] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 01:03:00 (1781586180) [ 2940.899278] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2940.905410] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2940.918549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2942.444658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2942.457048] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2942.477547] Lustre: Skipped 1 previous similar message [ 2944.288557] Lustre: server umount lustre-MDT0000 complete [ 2947.566741] LustreError: 94090:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2947.599364] LustreError: 94090:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2948.574985] LustreError: 94071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781586198 with bad export cookie 700376586109021083 [ 2948.593946] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2948.600559] LustreError: 94071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2949.079880] Lustre: server umount lustre-MDT0001 complete [ 2964.122951] Lustre: server umount lustre-OST0000 complete [ 2977.168317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2977.767434] LustreError: 98401:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2977.924576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2983.410566] LustreError: 98402:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2983.634201] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 2992.613251] LustreError: 98402:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2992.667583] LustreError: 98402:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2994.760862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2995.686083] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3001.415308] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 3009.095170] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3015.102229] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3030.523401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3030.739340] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 3030.746878] LustreError: 98388:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 3030.774676] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 3030.782680] LustreError: 100487:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -98 [ 3030.789470] LustreError: 100487:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 3030.794080] LustreError: 100487:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 3030.867395] Lustre: server umount lustre-OST0000 complete [ 3030.871774] LustreError: 100487:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 3040.028750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3040.190641] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 3040.199654] LustreError: 98390:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 3040.216486] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 3040.225964] LustreError: 100948:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -98 [ 3040.230644] LustreError: 100948:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 3040.239286] LustreError: 100948:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 3040.335252] Lustre: server umount lustre-OST0000 complete [ 3040.338495] LustreError: 100948:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 3050.943374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3051.142699] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3051.150655] Lustre: Found index 0 for lustre-OST0000, updating log [ 3051.157738] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 3051.275604] Lustre: lustre-OST0000: new disk, initializing [ 3051.284542] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3051.300544] Lustre: 101482:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3051.687442] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3059.422517] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 3064.012462] Lustre: Failing over lustre-OST0000 [ 3064.221618] Lustre: server umount lustre-OST0000 complete [ 3070.443353] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3070.463830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3070.466423] Lustre: Skipped 1 previous similar message [ 3070.479930] Lustre: Skipped 3 previous similar messages [ 3074.490139] Lustre: server umount lustre-MDT0000 complete [ 3075.554683] LustreError: 98402:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3075.590483] LustreError: 98402:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3078.809988] LustreError: 99144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781586328 with bad export cookie 700376586109023827 [ 3078.818436] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3078.830639] LustreError: 99144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3079.259700] Lustre: server umount lustre-MDT0001 complete [ 3090.007584] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 01:05:37 (1781586337) [ 3111.497178] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 3123.844894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3124.420559] LustreError: 103828:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3124.673480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3129.874571] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 3140.008429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3145.359436] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 3148.833964] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 3156.059670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3156.794093] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3156.799627] Lustre: Skipped 1 previous similar message [ 3163.981574] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 3173.833417] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3173.969201] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3174.044966] Lustre: lustre-OST0001: new disk, initializing [ 3174.047960] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3174.053618] Lustre: 106147:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3181.116890] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 3182.209863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3182.217305] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3182.312404] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3189.434670] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3194.241903] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3199.604299] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.49@tcp (stopping) [ 3202.017988] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 3202.024071] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3204.685408] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.49@tcp (stopping) [ 3204.694341] Lustre: Skipped 4 previous similar messages [ 3204.923036] Lustre: server umount lustre-MDT0001 complete [ 3208.173566] LustreError: 103825:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3208.196184] LustreError: 103825:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 3215.152378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3215.324431] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3215.330234] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3215.387334] Lustre: 103816:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3216.596972] LustreError: 107682:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3216.606241] LustreError: 107682:0:(tgt_mount.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3216.617223] LustreError: 103821:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3216.736992] LustreError: 107693:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.206.149@tcp: cfg command failed: rc = -17 [ 3216.749883] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3216.749883] [ 3221.095139] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 3221.484367] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3230.975948] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 01:07:58 (1781586478) [ 3249.896337] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 3270.254506] Lustre: 109767:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3296.865969] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3300.908888] Lustre: 110902:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3400.162103] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3400.172742] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3400.204902] Lustre: Skipped 2 previous similar messages [ 3400.216597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3400.679261] LustreError: 103823:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3400.711960] LustreError: 103823:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 3400.912866] Lustre: server umount lustre-MDT0000 complete [ 3405.719054] LustreError: 104570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781586655 with bad export cookie 700376586109024751 [ 3405.719736] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3405.727274] LustreError: 104570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3406.137668] Lustre: server umount lustre-MDT0001 complete [ 3413.738642] Lustre: server umount lustre-OST0000 complete [ 3418.515584] Lustre: server umount lustre-OST0001 complete [ 3425.009448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3436.114816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3447.711478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3462.427598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3492.301540] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 3506.973546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3506.991791] Lustre: lustre-MDT0000: reset Object Index mappings [ 3507.809605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3507.816895] Lustre: Skipped 2 previous similar messages [ 3512.628601] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 3523.087415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3523.105481] Lustre: lustre-MDT0001: reset Object Index mappings [ 3529.967591] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 3533.725116] Lustre: 116010:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3542.838352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3549.183882] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 3551.950982] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 3552.279972] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 3563.527785] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3569.676866] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3569.682813] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3574.176825] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 3586.578948] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3591.963302] Lustre: 117896:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3665.106032] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 01:15:11 (1781586911) [ 3692.518148] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3692.542441] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3692.548595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3692.548607] Lustre: Skipped 2 previous similar messages [ 3692.602214] Lustre: Skipped 8 previous similar messages [ 3694.858925] Lustre: server umount lustre-MDT0000 complete [ 3697.647669] LustreError: 117925:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3697.702475] LustreError: 117925:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 16 previous similar messages [ 3699.930697] LustreError: 114841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781586949 with bad export cookie 700376586109071623 [ 3699.943129] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3699.951567] LustreError: 114841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3706.669635] Lustre: server umount lustre-MDT0001 complete [ 3726.304331] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3726.535297] Lustre: server umount lustre-OST0000 complete [ 3736.911132] Lustre: server umount lustre-OST0001 complete [ 3744.507688] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_hostid [ 3757.719209] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 3807.160530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3807.725879] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3807.789486] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3807.940789] Lustre: lustre-MDT0000: new disk, initializing [ 3808.074929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3808.087200] Lustre: Skipped 3 previous similar messages [ 3808.120902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3814.281343] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 3823.134243] Lustre: server umount lustre-MDT0000 complete [ 3835.319702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3835.771444] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3835.783889] Lustre: 122178:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3835.792473] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3842.003635] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 3848.664823] Lustre: server umount lustre-MDT0000 complete [ 3861.223088] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 01:18:27 (1781587107) [ 3863.180734] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3865.996777] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 01:18:33 (1781587113) [ 3891.308204] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 3906.307691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3914.035796] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 3925.549616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3925.852458] Lustre: 124252:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3925.969158] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3925.983588] Lustre: Skipped 1 previous similar message [ 3926.167645] Lustre: lustre-MDT0001: new disk, initializing [ 3926.493242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3926.517483] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3933.226951] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 3940.137799] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3950.836157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3951.490483] Lustre: lustre-OST0000: new disk, initializing [ 3951.505957] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3951.519357] Lustre: 125832:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3953.376792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3953.393234] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3953.701613] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3961.106627] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 3974.045227] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3974.420149] Lustre: lustre-OST0001: new disk, initializing [ 3974.433560] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3974.440083] Lustre: 126702:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3976.529115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3976.547858] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3976.744748] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3984.100772] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 3996.582725] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4001.500806] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4517.492390] Lustre: 133136:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4517.499211] Lustre: 133136:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4517.505281] Lustre: 133136:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4517.513690] Lustre: 133136:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4517.521291] Lustre: 133136:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4517.530563] Lustre: 133136:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4894.293404] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 01:35:41 (1781588141) [ 4910.049568] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4910.065280] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4910.085397] Lustre: Skipped 2 previous similar messages [ 4910.098372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4910.106945] Lustre: Skipped 7 previous similar messages [ 4916.406223] Lustre: server umount lustre-MDT0000 complete [ 4916.740771] LustreError: 127509:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4916.771790] LustreError: 127509:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 4920.742763] LustreError: 124244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781588170 with bad export cookie 700376586109144500 [ 4920.745607] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4920.761190] LustreError: 124244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4921.825840] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4921.847642] Lustre: Skipped 3 previous similar messages [ 4921.857703] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4921.865417] Lustre: Skipped 5 previous similar messages [ 4931.053354] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4931.070608] Lustre: Skipped 3 previous similar messages [ 4935.138965] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4935.733055] Lustre: server umount lustre-MDT0001 complete [ 4946.619608] Lustre: server umount lustre-OST0000 complete [ 4965.344518] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4965.617655] Lustre: server umount lustre-OST0001 complete [ 4974.831210] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_hostid [ 4987.897086] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 5035.783971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5036.152807] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5036.207990] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5036.380523] Lustre: lustre-MDT0000: new disk, initializing [ 5036.551214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5036.564103] Lustre: Skipped 5 previous similar messages [ 5036.618452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5043.242967] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 5057.087165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5057.277847] Lustre: 136212:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5057.358597] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5057.362385] Lustre: Skipped 1 previous similar message [ 5057.427227] Lustre: lustre-MDT0001: new disk, initializing [ 5057.627257] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5057.655327] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5063.381532] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 5074.995884] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5082.219166] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5089.819269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5090.117063] Lustre: lustre-OST0000: new disk, initializing [ 5090.121137] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5090.127639] Lustre: 138182:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5090.209733] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5090.220646] Lustre: Skipped 1 previous similar message [ 5091.314695] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5091.334561] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5091.531988] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5098.039469] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 5108.537198] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5119.872225] Lustre: lustre-MDD0000: changelog on [ 5122.887193] Lustre: lustre-MDD0001: changelog on [ 7365.555438] Lustre: lustre-MDD0001: changelog off [ 7367.743177] Lustre: 139996:0:(llog_cat.c:977:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 7368.784557] Lustre: lustre-MDD0000: changelog off [ 7379.952354] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7379.957748] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7379.959025] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7379.959833] Lustre: Skipped 2 previous similar messages [ 7385.079451] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7385.090752] Lustre: Skipped 2 previous similar messages [ 7390.179851] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7390.186315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7390.207141] Lustre: Skipped 1 previous similar message [ 7390.475574] Lustre: server umount lustre-OST0000 complete [ 7397.345666] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7397.356225] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7400.437633] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7400.442198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7400.464627] Lustre: Skipped 1 previous similar message [ 7402.155779] Lustre: server umount lustre-MDT0000 complete [ 7405.561839] LustreError: 137091:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7405.582356] LustreError: 137091:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 7406.747456] LustreError: 136203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781590656 with bad export cookie 700376586109971193 [ 7406.752798] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7406.764159] LustreError: 136203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7407.237236] Lustre: server umount lustre-MDT0001 complete [ 7420.591377] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [ 7423.813777] Key type lgssc unregistered [ 7424.143907] LNet: 147391:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7424.156418] LNetError: 147391:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7424.177136] LNet: Removed LNI 192.168.206.149@tcp [ 7425.421240] Key type .llcrypt unregistered [ 7425.425540] Key type ._llcrypt unregistered [ 7447.813544] Key type ._llcrypt registered [ 7447.816048] Key type .llcrypt registered [ 7447.920564] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_hostid [ 7464.515851] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 7465.733463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7465.897356] alg: No test for adler32 (adler32-zlib) [ 7467.087987] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 7467.405804] LNet: Added LNI 192.168.206.149@tcp [8/256/0/180] [ 7469.091155] Key type lgssc registered [ 7470.380384] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7513.168837] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7513.199450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7514.607184] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7514.654347] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7514.743764] Lustre: lustre-MDT0000: new disk, initializing [ 7514.830238] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7514.845664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7520.729687] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 7534.848265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7535.001614] Lustre: 150592:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7535.049718] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7535.055712] Lustre: Skipped 1 previous similar message [ 7535.164794] Lustre: lustre-MDT0001: new disk, initializing [ 7535.267522] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7535.304518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7535.321759] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7541.064750] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 7551.430941] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7558.657645] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7566.441929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7566.751489] Lustre: lustre-OST0000: new disk, initializing [ 7566.756874] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7566.764362] Lustre: 152560:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7566.896649] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7573.674788] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 7576.679242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7576.698715] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7576.838452] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7585.129753] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7591.495644] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_module obdecho/obdecho [ 7593.511592] LustreError: 153563:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 7593.517591] LustreError: 153563:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 7595.397810] LustreError: 153659:0:(echo_client.c:1883:echo_md_handler()) Can not resolve the path : rc = -2 [ 7612.391289] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7612.401832] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7612.428845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7612.902388] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7612.922631] Lustre: Skipped 1 previous similar message [ 7612.938617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7612.950625] Lustre: Skipped 1 previous similar message [ 7614.999376] Lustre: server umount lustre-MDT0000 complete [ 7619.232854] LustreError: 150583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781590869 with bad export cookie 13397637654455173468 [ 7619.240607] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7619.257144] LustreError: 150583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7619.697267] Lustre: server umount lustre-MDT0001 complete [ 7634.459623] Lustre: server umount lustre-OST0000 complete [ 7658.135292] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_hostid [ 7670.541573] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 7714.110524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7714.614172] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7714.655655] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7714.775639] Lustre: lustre-MDT0000: new disk, initializing [ 7714.887150] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7714.904590] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7721.726982] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 7735.603022] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7735.704245] Lustre: 157382:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7735.756809] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7735.759985] Lustre: Skipped 1 previous similar message [ 7735.834056] Lustre: lustre-MDT0001: new disk, initializing [ 7735.964704] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7735.991655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7735.997931] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7741.562545] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 7751.471967] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7758.204454] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7765.510488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7765.769862] Lustre: lustre-OST0000: new disk, initializing [ 7765.774257] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7765.779697] Lustre: 159349:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7765.931797] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7771.715552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7771.731397] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7771.921318] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7773.357711] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 7784.900735] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7791.058605] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7791.362363] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7797.170713] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7797.471712] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7802.859162] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7802.860240] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7802.868190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7802.876691] LustreError: Skipped 1 previous similar message [ 7802.894926] Lustre: Skipped 1 previous similar message [ 7807.978495] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7807.984790] Lustre: Skipped 1 previous similar message [ 7813.094857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7813.105672] Lustre: Skipped 1 previous similar message [ 7814.112185] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7814.287828] Lustre: server umount lustre-OST0000 complete [ 7822.821889] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7822.839816] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7822.875922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7822.883759] Lustre: Skipped 1 previous similar message [ 7825.195511] Lustre: server umount lustre-MDT0000 complete [ 7828.451620] LustreError: 157388:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7828.473990] LustreError: 157388:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 7829.891540] LustreError: 157374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781591079 with bad export cookie 13397637654455174427 [ 7829.893935] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7829.899574] LustreError: 157374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7830.575518] Lustre: server umount lustre-MDT0001 complete [ 7835.096955] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 02:24:42 (1781591082) [ 7837.249179] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7839.458967] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 02:24:47 (1781591087) [ 7848.852662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7849.613384] LustreError: 161764:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7849.713622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7855.074682] LustreError: 161765:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7855.645499] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 7860.205867] LustreError: 161764:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7865.315653] LustreError: 161765:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7866.043394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7872.204335] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 7880.895125] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7888.327392] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7895.493590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7895.941518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7895.947706] Lustre: Skipped 1 previous similar message [ 7903.902608] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 7911.789540] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7930.854058] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7930.868055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7930.884646] Lustre: Skipped 3 previous similar messages [ 7930.885613] Lustre: Skipped 3 previous similar messages [ 7935.377713] Lustre: server umount lustre-MDT0000 complete [ 7935.979655] LustreError: 161765:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7935.991757] LustreError: 161765:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7939.960116] LustreError: 161746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781591189 with bad export cookie 13397637654455175309 [ 7939.971275] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7939.973854] LustreError: 161746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7941.099075] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7953.492619] LustreError: 161761:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7953.510623] LustreError: 161761:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 7954.402052] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7954.782811] Lustre: server umount lustre-MDT0001 complete [ 7961.530034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7975.651324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7976.108555] LustreError: 165633:0:(llog.c:1646:llog_backup()) MGC192.168.206.149@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7976.116921] Lustre: 165633:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.149@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7990.247045] LustreError: 165640:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7990.264388] LustreError: 165640:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 8037.088381] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 8037.094539] Lustre: 165633:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.206.149@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 8055.266384] LustreError: 165656:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8055.288750] LustreError: 165656:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 47 previous similar messages [ 8077.816794] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8082.933645] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8083.754355] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 8095.037574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8105.449072] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 8115.189679] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8115.435838] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 8124.702711] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8126.652715] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8133.008937] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8135.228593] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8161.250720] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8161.266850] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8161.273897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8161.280756] Lustre: Skipped 2 previous similar messages [ 8161.306472] Lustre: Skipped 11 previous similar messages [ 8163.451320] Lustre: server umount lustre-MDT0000 complete [ 8167.857726] LustreError: 164573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781591417 with bad export cookie 13397637654455181798 [ 8167.862235] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8167.866732] LustreError: 164573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8168.210183] Lustre: server umount lustre-MDT0001 complete [ 8182.842374] Lustre: server umount lustre-OST0000 complete [ 8206.644137] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_hostid [ 8218.136881] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 8261.979420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8262.425306] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8262.471784] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8262.557163] Lustre: lustre-MDT0000: new disk, initializing [ 8262.635894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8262.649034] Lustre: Skipped 1 previous similar message [ 8262.678063] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8267.841859] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 8278.832886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8278.977530] Lustre: 170746:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8279.043877] LustreError: 170757:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8279.073569] LustreError: 170757:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 46 previous similar messages [ 8279.306649] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8284.244634] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 8295.082728] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8300.933697] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8307.895939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8308.159611] Lustre: lustre-OST0000: new disk, initializing [ 8308.161659] Lustre: Skipped 1 previous similar message [ 8308.168172] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8308.171898] Lustre: Skipped 2 previous similar messages [ 8308.177780] Lustre: 172714:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8314.288187] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 8315.964459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8315.976171] Lustre: Skipped 1 previous similar message [ 8315.983081] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8316.077943] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8323.726357] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8329.383942] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8329.760842] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8334.843978] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8335.143749] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8341.994015] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8341.995257] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8342.002033] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8342.034348] Lustre: Skipped 1 previous similar message [ 8343.540738] Lustre: server umount lustre-OST0000 complete [ 8353.827627] Lustre: server umount lustre-MDT0000 complete [ 8358.005712] LustreError: 170736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781591607 with bad export cookie 13397637654455198605 [ 8358.011045] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8358.014072] LustreError: 170736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8358.438632] Lustre: server umount lustre-MDT0001 complete [ 8368.296875] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 02:33:35 (1781591615) [ 8377.092871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8385.145097] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 8399.007909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8406.608620] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 8417.660478] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8424.980220] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8433.509793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8442.281444] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 8451.543377] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8462.768381] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 02:35:10 (1781591710) [ 8467.938446] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8467.949116] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8467.951111] LustreError: Skipped 2 previous similar messages [ 8467.973383] Lustre: Skipped 3 previous similar messages [ 8472.871656] Lustre: server umount lustre-OST0000 complete [ 8478.186715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8478.196659] Lustre: Skipped 6 previous similar messages [ 8484.161640] Lustre: server umount lustre-MDT0000 complete [ 8488.007771] LustreError: 175105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781591737 with bad export cookie 13397637654455199487 [ 8488.019448] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8488.438966] Lustre: server umount lustre-MDT0001 complete [ 8500.425802] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [ 8503.739570] Key type lgssc unregistered [ 8504.062811] LNet: 178853:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8504.070138] LNetError: 178853:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8504.089776] LNet: Removed LNI 192.168.206.149@tcp [ 8505.023877] Key type .llcrypt unregistered [ 8505.026154] Key type ._llcrypt unregistered [ 8527.056254] Key type ._llcrypt registered [ 8527.058034] Key type .llcrypt registered [ 8527.214429] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 8529.141601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8529.219230] alg: No test for adler32 (adler32-zlib) [ 8530.441386] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 8530.754380] LNet: Added LNI 192.168.206.149@tcp [8/256/0/180] [ 8532.472395] Key type lgssc registered [ 8533.906862] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8542.600056] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8542.635951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8558.240540] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8563.360253] LustreError: 180102:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.149@tcp: failed processing log, type 4: rc = -110 [ 8568.480547] LustreError: 180102:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8568.497593] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 8568.504511] LustreError: MGC192.168.206.149@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 8568.513800] LustreError: 180102:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-OST0000: -22 [ 8568.521789] LustreError: 180102:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -22 [ 8568.531560] LustreError: 180102:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 8568.621936] Lustre: server umount lustre-OST0000 complete [ 8568.628603] LustreError: 180102:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8578.351354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8579.863357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8584.575362] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 8589.285804] LustreError: 180602:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8589.330465] LustreError: 180602:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8593.722223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8594.251534] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8600.241669] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 8610.027839] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8616.151165] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8625.909407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8626.304160] LustreError: 182421:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8626.320536] Lustre: 182421:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.206.149@tcp: invalid local config log lustre-OST0000: rc = -22 [ 8626.489955] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8635.516472] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 8646.880114] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8650.722404] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8650.732693] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8650.747893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8654.314417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8654.322825] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8655.755779] Lustre: server umount lustre-MDT0000 complete [ 8659.357563] LustreError: 180583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781591909 with bad export cookie 294532874310777072 [ 8659.365909] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8659.374801] LustreError: 180583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8659.428504] LustreError: 180599:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8659.431239] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8659.451716] Lustre: Skipped 1 previous similar message [ 8659.456653] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8659.459398] Lustre: Skipped 2 previous similar messages [ 8659.685369] Lustre: server umount lustre-MDT0001 complete [ 8666.127242] Lustre: server umount lustre-OST0000 complete [ 8675.694922] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_hostid [ 8687.308562] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 8729.509145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8729.869648] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8729.919687] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8730.021859] Lustre: lustre-MDT0000: new disk, initializing [ 8730.151530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8730.169803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8735.220432] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 8746.606157] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8746.708826] Lustre: 185945:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8746.792604] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8746.801314] Lustre: Skipped 1 previous similar message [ 8746.946943] Lustre: lustre-MDT0001: new disk, initializing [ 8747.068513] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8747.119057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8747.138027] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8752.617377] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 8763.621949] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8770.508155] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8779.169487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8779.368948] Lustre: lustre-OST0000: new disk, initializing [ 8779.373319] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8779.377592] Lustre: 187916:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8779.458679] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8785.076430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8785.094301] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8785.247799] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8786.671438] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 8798.785710] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8804.399255] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8804.828146] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8810.937833] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8811.248544] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8816.109228] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8816.109815] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8816.130479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8816.130839] LustreError: Skipped 1 previous similar message [ 8816.154891] Lustre: Skipped 1 previous similar message [ 8819.693107] Lustre: server umount lustre-OST0000 complete [ 8826.338545] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8826.348058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8826.354875] Lustre: Skipped 2 previous similar messages [ 8826.371911] Lustre: Skipped 1 previous similar message [ 8831.313906] Lustre: server umount lustre-MDT0000 complete [ 8831.462373] LustreError: 185956:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8831.489382] LustreError: 185956:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8836.242755] LustreError: 185938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781592086 with bad export cookie 294532874310777786 [ 8836.265210] LustreError: 185938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8836.266558] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8836.643720] Lustre: server umount lustre-MDT0001 complete [ 8848.634719] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 02:41:35 (1781592095) [ 8873.520262] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [ 8877.114245] Key type lgssc unregistered [ 8877.462558] LNet: 190862:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8877.474814] LNetError: 190862:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8878.502938] LNet: Removed LNI 192.168.206.149@tcp [ 8879.516461] Key type .llcrypt unregistered [ 8879.518897] Key type ._llcrypt unregistered [ 8901.418434] Key type ._llcrypt registered [ 8901.422724] Key type .llcrypt registered [ 8901.565843] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 8902.737159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8902.795115] alg: No test for adler32 (adler32-zlib) [ 8904.130730] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [ 8904.442357] LNet: Added LNI 192.168.206.149@tcp [8/256/0/180] [ 8906.225449] Key type lgssc registered [ 8907.647364] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8918.415919] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8918.458585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8920.113611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8926.196748] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 8929.253372] LustreError: 192086:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8929.274970] LustreError: 192086:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8934.371214] LustreError: 192085:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8937.767447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8938.412439] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8944.248215] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 8953.722400] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8961.225378] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8969.629695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8969.948837] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8969.956260] LustreError: 193951:0:(llog.c:1687:llog_backup()) MGC192.168.206.149@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8969.965898] Lustre: 193951:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.206.149@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8970.101681] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8977.427171] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 8984.345918] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8988.644043] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8988.645975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8988.659033] Lustre: Skipped 2 previous similar messages [ 8988.668717] Lustre: Skipped 2 previous similar messages [ 8993.055944] Lustre: server umount lustre-MDT0000 complete [ 8993.764832] LustreError: 192081:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8993.777167] LustreError: 192081:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8997.281605] LustreError: 192067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781592247 with bad export cookie 14354573907335202198 [ 8997.299152] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8997.299247] LustreError: 192067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8997.750933] Lustre: server umount lustre-MDT0001 complete [ 9012.860311] Lustre: server umount lustre-OST0000 complete [ 9023.278383] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_hostid [ 9035.555528] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 9083.584348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9083.863201] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9083.921957] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9084.049659] Lustre: lustre-MDT0000: new disk, initializing [ 9084.190371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9084.223087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9089.322488] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 9101.083314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9101.225631] Lustre: 197478:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9101.271730] Lustre: *** cfs_fail_loc=131e, val=0*** [ 9101.274096] Lustre: Skipped 1 previous similar message [ 9101.277115] LustreError: 198261:0:(llog.c:1687:llog_backup()) MGC192.168.206.149@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 9101.284832] LustreError: 198261:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 9101.292201] Lustre: 198261:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.149@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 9101.310970] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9101.318820] Lustre: Skipped 1 previous similar message [ 9101.407818] Lustre: lustre-MDT0001: new disk, initializing [ 9101.591514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9101.648192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9101.660200] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9107.420818] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 9119.538515] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9126.454923] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9135.388510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9135.710193] Lustre: *** cfs_fail_loc=131e, val=0*** [ 9135.717869] Lustre: Skipped 1 previous similar message [ 9135.735702] LustreError: 199422:0:(llog.c:1687:llog_backup()) MGC192.168.206.149@tcp: failed to backup log lustre-OST0000: rc = -28 [ 9135.751500] LustreError: 199422:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 9135.763177] Lustre: 199422:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.149@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 9135.775260] Lustre: 199422:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 9135.795340] Lustre: lustre-OST0000: new disk, initializing [ 9135.800220] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9135.818731] Lustre: 199445:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9135.994353] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9144.438453] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9144.462365] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9144.516215] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 9144.678779] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9155.111201] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9161.228964] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9161.557874] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9166.450332] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9166.697147] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9170.402194] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9170.408412] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9170.412065] LustreError: Skipped 1 previous similar message [ 9170.438891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9170.444776] Lustre: Skipped 1 previous similar message [ 9174.861114] Lustre: server umount lustre-OST0000 complete [ 9180.641742] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9180.649305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9180.660866] Lustre: Skipped 2 previous similar messages [ 9180.680366] Lustre: Skipped 1 previous similar message [ 9185.357656] Lustre: server umount lustre-MDT0000 complete [ 9185.766746] LustreError: 197489:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9185.795979] LustreError: 197489:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9190.328492] LustreError: 197468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781592440 with bad export cookie 14354573907335202912 [ 9190.329479] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9190.341892] LustreError: 197468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9190.718957] Lustre: server umount lustre-MDT0001 complete [ 9200.688418] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 02:47:28 (1781592448) [ 9220.822689] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 9232.826534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9233.360129] LustreError: 202577:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9233.408963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9239.546402] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 9243.646273] LustreError: 202578:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9243.673782] LustreError: 202578:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9250.120469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9256.231182] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 9259.921613] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9266.972566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9267.377323] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9267.385104] Lustre: Skipped 1 previous similar message [ 9274.269714] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 9285.430996] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9285.702762] Lustre: lustre-OST0001: new disk, initializing [ 9285.713130] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9285.726698] Lustre: 204887:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9291.963684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9291.981642] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9292.252618] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9294.776512] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 9306.293670] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9317.343650] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9319.688778] Lustre: DEBUG MARKER: ADD OST3 [ 9325.278947] loop: module loaded [ 9325.357759] loop0: detected capacity change from 0 to 204800000 [ 9325.633385] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9325.639068] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9325.649244] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9332.999484] loop0: detected capacity change from 0 to 204800000 [ 9333.458259] Lustre: DEBUG MARKER: STOP OST3 [ 9340.051659] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9340.272770] Lustre: lustre-OST0003: new disk, initializing [ 9340.280638] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 9340.290197] Lustre: 206927:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9340.449068] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 9340.458275] Lustre: Skipped 1 previous similar message [ 9342.274649] Lustre: *** cfs_fail_loc=2109, val=2*** [ 9342.420397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 9342.426720] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 9342.475646] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 9344.157405] Lustre: Failing over lustre-OST0003 [ 9344.290267] LustreError: 207076:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 9344.437110] Lustre: server umount lustre-OST0003 complete [ 9347.161488] LustreError: 204063:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.206.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9347.175350] LustreError: 204063:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9347.566858] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9347.577332] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9347.606246] Lustre: Skipped 1 previous similar message [ 9348.595263] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 9357.716464] Lustre: DEBUG MARKER: START OST3 again [ 9366.166206] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9366.739481] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 9367.628549] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 9367.925196] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 9367.926775] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 9373.980669] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 9396.166755] Lustre: Failing over lustre-OST0003 [ 9396.328242] Lustre: server umount lustre-OST0003 complete [ 9397.751103] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9397.754192] LustreError: 205136:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9397.772881] Lustre: Skipped 1 previous similar message [ 9397.800915] LustreError: 205136:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 9418.220867] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9418.222266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9418.245039] Lustre: Skipped 3 previous similar messages [ 9418.284919] Lustre: Skipped 3 previous similar messages [ 9421.455116] Lustre: server umount lustre-MDT0000 complete [ 9425.943841] LustreError: 206924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781592675 with bad export cookie 14354573907335203794 [ 9425.948948] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9425.958244] LustreError: 206924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9428.456641] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9428.465668] Lustre: Skipped 1 previous similar message [ 9432.555497] Lustre: server umount lustre-MDT0001 complete [ 9437.471103] Lustre: server umount lustre-OST0000 complete [ 9442.390652] Lustre: server umount lustre-OST0001 complete [ 9450.008072] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_hostid [ 9459.715587] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 9509.079112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9509.431788] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9509.482905] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9509.624946] Lustre: lustre-MDT0000: new disk, initializing [ 9509.746300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9509.751075] Lustre: Skipped 1 previous similar message [ 9509.769644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9515.202795] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 9529.302299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9529.552847] Lustre: 212348:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9529.623551] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9529.631211] Lustre: Skipped 1 previous similar message [ 9529.750572] Lustre: lustre-MDT0001: new disk, initializing [ 9529.902638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9529.923374] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9535.920574] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 9546.063301] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9551.686096] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9559.038068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9559.448964] Lustre: 214318:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9560.552988] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9560.738716] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9566.114476] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 9575.167907] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9580.086712] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9580.368848] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9586.573754] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9586.972781] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9591.266980] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9591.269057] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9591.296504] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9591.303478] Lustre: Skipped 3 previous similar messages [ 9591.340115] Lustre: Skipped 3 previous similar messages [ 9595.902857] Lustre: server umount lustre-OST0000 complete [ 9601.507616] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9607.504443] Lustre: server umount lustre-MDT0000 complete [ 9611.745717] LustreError: 212357:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9611.760475] LustreError: 212357:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 17 previous similar messages [ 9611.947073] LustreError: 213155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781592861 with bad export cookie 14354573907335206090 [ 9611.950569] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9611.963025] LustreError: 213155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9612.574624] Lustre: server umount lustre-MDT0001 complete [ 9617.167927] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 02:54:24 (1781592864) [ 9637.551880] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_hostid [ 9648.835429] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [ 9687.680923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9687.942857] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9687.998874] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9688.006413] Lustre: Skipped 1 previous similar message [ 9688.081585] Lustre: lustre-MDT0000: new disk, initializing [ 9688.084875] Lustre: Skipped 1 previous similar message [ 9688.206435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9688.212200] Lustre: Skipped 1 previous similar message [ 9693.360667] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 9706.336373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9706.625170] Lustre: 218929:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9707.140311] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9712.737137] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 9725.168956] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9732.368510] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9741.536355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9742.113557] Lustre: 220901:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9743.484508] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9743.708680] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9751.052787] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 9761.768645] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9767.688887] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9767.923417] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9773.732855] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9773.947714] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9779.693673] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9779.701095] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9779.702164] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9779.702172] Lustre: Skipped 4 previous similar messages [ 9779.709650] Lustre: Skipped 3 previous similar messages [ 9779.748338] LustreError: Skipped 1 previous similar message [ 9782.751728] Lustre: server umount lustre-OST0000 complete [ 9788.899782] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9801.187138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9801.451726] Lustre: server umount lustre-MDT0000 complete [ 9805.299713] LustreError: 218941:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9805.315087] LustreError: 218941:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9805.680605] LustreError: 218922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781593055 with bad export cookie 14354573907335206972 [ 9805.692141] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9805.697162] LustreError: 218922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9806.126939] Lustre: server umount lustre-MDT0001 complete [ 9815.306305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9815.815565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9815.827409] Lustre: Skipped 5 previous similar messages [ 9820.760059] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 9830.061377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9835.318497] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 9843.824231] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9850.049717] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9857.828059] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9865.768810] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [ 9873.681430] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9886.187058] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9886.205760] Lustre: Skipped 3 previous similar messages [ 9886.207483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9886.231922] Lustre: Skipped 9 previous similar messages [ 9890.833300] Lustre: server umount lustre-OST0000 complete [ 9901.881379] Lustre: server umount lustre-MDT0000 complete [ 9905.995711] LustreError: 223103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781593155 with bad export cookie 14354573907335207854 [ 9906.006344] LustreError: 223103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9906.008631] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9906.266302] Lustre: server umount lustre-MDT0001 complete [10013.132215] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [10016.478764] Key type lgssc unregistered [10016.851534] LNet: 226797:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10016.868660] LNetError: 226797:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10016.885465] LNet: Removed LNI 192.168.206.149@tcp [10018.231184] Key type .llcrypt unregistered [10018.233067] Key type ._llcrypt unregistered [10044.981470] Key type ._llcrypt registered [10044.986338] Key type .llcrypt registered [10045.081239] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_hostid [10062.014375] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [10063.537168] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10063.591631] alg: No test for adler32 (adler32-zlib) [10064.961926] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [10065.378319] LNet: Added LNI 192.168.206.149@tcp [8/256/0/180] [10067.192194] Key type lgssc registered [10068.576962] Lustre: Echo OBD driver; http://www.lustre.org/ [10109.206386] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10109.255279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10110.524305] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10110.555726] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10110.634150] Lustre: lustre-MDT0000: new disk, initializing [10110.757496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10110.774818] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10115.715415] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [10127.374306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10127.549528] Lustre: 230152:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10127.669404] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10127.675885] Lustre: Skipped 1 previous similar message [10127.798821] Lustre: lustre-MDT0001: new disk, initializing [10127.906857] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10127.965091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10127.989434] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10132.552248] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [10142.138676] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10148.084711] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10155.442934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10155.827735] Lustre: lustre-OST0000: new disk, initializing [10155.842815] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10155.850037] Lustre: 232121:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10155.975858] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10162.262888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10162.282605] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10162.390335] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10162.639862] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [10172.170071] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10177.338388] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10177.539992] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10182.645376] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10182.892518] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10188.260535] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10188.260753] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10188.269173] LustreError: Skipped 1 previous similar message [10188.277379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10188.295371] Lustre: Skipped 1 previous similar message [10193.382099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10193.389834] Lustre: Skipped 1 previous similar message [10198.500441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10199.008309] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10199.231939] Lustre: server umount lustre-OST0000 complete [10204.641606] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10204.651722] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10204.665622] Lustre: Skipped 1 previous similar message [10204.674414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10204.686892] Lustre: Skipped 1 previous similar message [10208.737272] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10208.742499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10208.753244] Lustre: Skipped 1 previous similar message [10210.914439] Lustre: server umount lustre-MDT0000 complete [10215.250082] LustreError: 230143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781593465 with bad export cookie 15742427406935137098 [10215.259159] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10215.261908] LustreError: 230143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10215.578850] Lustre: server umount lustre-MDT0001 complete [10235.807548] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [10247.424501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10248.150126] LustreError: 235056:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10248.193158] LustreError: 235056:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10248.232981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10253.286853] LustreError: 235057:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10253.557974] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [10257.381549] LustreError: 235056:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10262.364035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10262.503523] LustreError: 235057:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10262.811461] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10268.249747] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [10271.029794] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10278.455279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10278.851818] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10285.398908] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [10295.334082] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10295.643278] Lustre: lustre-OST0001: new disk, initializing [10295.647533] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10295.654654] Lustre: 237366:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10295.797194] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10301.624738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10301.648979] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10301.781301] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10303.160317] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [10316.151399] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10321.823659] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10336.297443] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 03:06:24 (1781593584) [10339.811062] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10339.828338] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10339.850852] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10345.348752] Lustre: server umount lustre-MDT0000 complete [10348.011160] LustreError: 236024:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10348.050776] LustreError: 236024:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10349.090752] LustreError: 235790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781593598 with bad export cookie 15742427406935137980 [10349.094211] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10349.112822] LustreError: 235790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10349.637477] Lustre: server umount lustre-MDT0001 complete [10364.266590] Lustre: server umount lustre-OST0000 complete [10368.480630] Lustre: 228375:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781593602/real 1781593602] req@ffff8cb88b78bb80 x1868135991986560/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781593618 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10368.516508] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10368.526773] Lustre: Skipped 3 previous similar messages [10369.080590] Lustre: server umount lustre-OST0001 complete [10376.924751] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_hostid [10389.847681] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [10434.370759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10434.716979] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10434.763615] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10434.857548] Lustre: lustre-MDT0000: new disk, initializing [10435.004809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10435.029397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10439.672961] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [10451.758440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10451.922136] Lustre: 241319:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10452.023458] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10452.033091] Lustre: Skipped 1 previous similar message [10452.145321] Lustre: lustre-MDT0001: new disk, initializing [10452.309331] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10452.321035] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10457.682034] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [10469.085141] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10476.676884] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10485.974089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10486.411638] Lustre: lustre-OST0000: new disk, initializing [10486.415531] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10486.425196] Lustre: 243290:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10487.608584] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10487.637044] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10487.865297] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10496.341367] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [10513.585952] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10520.304690] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10520.655133] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10527.477906] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10527.738304] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10533.869626] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10533.880280] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10533.910883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10533.914187] Lustre: Skipped 4 previous similar messages [10536.209871] Lustre: server umount lustre-OST0000 complete [10555.361473] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10555.736341] Lustre: server umount lustre-MDT0000 complete [10558.435709] LustreError: 241324:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10558.466252] LustreError: 241324:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10560.907821] LustreError: 242125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781593810 with bad export cookie 15742427406935140115 [10560.919549] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10560.928480] LustreError: 242125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10561.765680] Lustre: server umount lustre-MDT0001 complete [10573.775442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10574.570466] LustreError: 245509:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10574.830798] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10574.849960] Lustre: Skipped 2 previous similar messages [10581.143304] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [10592.991352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10600.424036] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [10610.806957] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10618.197635] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10627.940843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10637.191254] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [10649.132820] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10797.542335] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10797.564239] LustreError: Skipped 1 previous similar message [10797.581601] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10797.605768] Lustre: Skipped 3 previous similar messages [10797.617913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10797.622717] Lustre: Skipped 6 previous similar messages [10803.753611] Lustre: server umount lustre-OST0000 complete [10823.136487] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10823.415644] Lustre: server umount lustre-MDT0000 complete [10824.168510] LustreError: 247923:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10824.192417] LustreError: 247923:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [10828.681873] LustreError: 247354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781594078 with bad export cookie 15742427406935140997 [10828.695576] LustreError: 247354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10828.700832] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10829.283339] Lustre: server umount lustre-MDT0001 complete [10919.027508] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [10922.696028] Key type lgssc unregistered [10923.108818] LNet: 249122:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10923.121439] LNetError: 249122:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10923.151969] LNet: Removed LNI 192.168.206.149@tcp [10924.323248] Key type .llcrypt unregistered [10924.325077] Key type ._llcrypt unregistered [10962.204922] Key type ._llcrypt registered [10962.206815] Key type .llcrypt registered [10962.384697] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_hostid [10982.750128] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [10985.339753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10985.451944] alg: No test for adler32 (adler32-zlib) [10986.945501] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [10987.304854] LNet: Added LNI 192.168.206.149@tcp [8/256/0/180] [10989.048202] Key type lgssc registered [10991.763729] Lustre: Echo OBD driver; http://www.lustre.org/ [11054.998140] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [11070.063752] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11070.105714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11071.476952] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11071.504985] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11071.605530] Lustre: lustre-MDT0000: new disk, initializing [11071.808425] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11071.833749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11077.674451] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [11092.297377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11092.442428] Lustre: 253224:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11092.552360] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11092.557903] Lustre: Skipped 1 previous similar message [11092.630937] Lustre: lustre-MDT0001: new disk, initializing [11092.793372] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11092.906103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11092.921078] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11098.938605] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [11104.782305] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11113.809878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11114.284166] Lustre: lustre-OST0000: new disk, initializing [11114.289393] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11114.302696] Lustre: 254854:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11114.481962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11123.195954] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [11124.360727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11124.374131] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11124.595939] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11136.943795] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11137.289237] Lustre: lustre-OST0001: new disk, initializing [11137.294247] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11137.313072] Lustre: 255728:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11137.427560] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11145.402926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11145.415314] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11145.521407] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11145.992770] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [11160.339318] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11171.340207] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11186.660690] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11186.674676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11186.680207] Lustre: Skipped 3 previous similar messages [11186.701848] Lustre: Skipped 3 previous similar messages [11191.789472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11191.807342] Lustre: Skipped 3 previous similar messages [11192.232208] Lustre: server umount lustre-MDT0000 complete [11197.066222] LustreError: 253217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781594446 with bad export cookie 13050389051791032069 [11197.074081] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11197.078290] LustreError: 253217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11197.633168] Lustre: server umount lustre-MDT0001 complete [11213.668913] Lustre: server umount lustre-OST0000 complete [11217.889043] Lustre: 250706:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781594451/real 1781594451] req@ffff8cb88cb01500 x1868136958740096/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781594467 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11217.919935] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11219.294499] Lustre: server umount lustre-OST0001 complete [11239.057865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [11253.408789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [11264.848214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [11278.713458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [11308.673937] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [11325.837574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11325.882594] Lustre: lustre-MDT0000: reset Object Index mappings [11326.784816] LustreError: 260745:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11326.810058] LustreError: 260745:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [11326.904660] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11332.069811] LustreError: 260746:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11334.535510] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [11337.194684] LustreError: 260745:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11342.310679] LustreError: 260746:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11347.432144] LustreError: 260745:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11348.378495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11348.406192] Lustre: lustre-MDT0001: reset Object Index mappings [11349.057231] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11356.976528] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [11361.377408] Lustre: 261890:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11373.296480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11373.609119] LustreError: 262245:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11373.932025] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11385.079459] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [11394.534222] LustreError: 262247:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11394.577050] LustreError: 262247:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [11397.650619] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [11400.671394] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11406.356206] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [11410.858118] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [11422.166915] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11428.131288] Lustre: 263766:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11437.027374] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11437.027374] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11437.067786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11442.148648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11442.168174] Lustre: Skipped 6 previous similar messages [11442.522491] Lustre: server umount lustre-MDT0000 complete [11447.277773] LustreError: 261481:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11447.297360] LustreError: 261481:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [11448.749129] LustreError: 260726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781594698 with bad export cookie 13050389051791038866 [11448.757223] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11448.774942] LustreError: 260726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11449.639970] Lustre: server umount lustre-MDT0001 complete [11465.820726] Lustre: server umount lustre-OST0000 complete [11468.773660] Lustre: 250705:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781594702/real 1781594702] req@ffff8cb98652b100 x1868136958802688/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781594718 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11468.844893] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11468.868667] Lustre: Skipped 2 previous similar messages [11472.509264] Lustre: server umount lustre-OST0001 complete [11501.053423] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 03:25:48 (1781594748) [11523.221934] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_hostid [11539.803375] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [11587.951106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11588.349391] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11588.383542] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11588.488299] Lustre: lustre-MDT0000: new disk, initializing [11588.605481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11588.610734] Lustre: Skipped 1 previous similar message [11588.632267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11595.011914] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [11609.271514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11609.524495] Lustre: 268045:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11609.656392] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11609.666343] Lustre: Skipped 1 previous similar message [11609.816254] Lustre: lustre-MDT0001: new disk, initializing [11610.021750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11610.042305] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11615.843864] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [11626.749336] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11633.452541] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11643.632298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11644.026678] Lustre: lustre-OST0000: new disk, initializing [11644.039644] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11644.049173] Lustre: 270018:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11645.789714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11645.798646] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11646.068818] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11653.142232] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [11665.005447] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11671.533455] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11671.806434] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11676.864716] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11677.119240] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11681.774369] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11681.781467] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11681.785523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11681.815750] Lustre: Skipped 1 previous similar message [11685.310950] Lustre: server umount lustre-OST0000 complete [11692.021911] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11692.022952] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11692.062713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11692.067045] LustreError: Skipped 1 previous similar message [11692.098963] Lustre: Skipped 2 previous similar messages [11696.840329] Lustre: server umount lustre-MDT0000 complete [11701.225674] LustreError: 268059:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11701.245754] LustreError: 268059:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11702.343968] LustreError: 270017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781594952 with bad export cookie 13050389051791040959 [11702.357765] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11702.358505] LustreError: 270017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11703.040904] Lustre: server umount lustre-MDT0001 complete [11731.894672] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [11746.284873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11747.090496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11747.101594] Lustre: Skipped 2 previous similar messages [11754.570631] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [11766.184294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11772.235437] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [11776.502718] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11786.041159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11796.049552] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [11807.404292] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11807.653282] Lustre: lustre-OST0001: new disk, initializing [11807.657853] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11807.667492] Lustre: 275289:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11809.799689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11809.814066] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11809.986703] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11817.690655] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [11830.542993] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11835.698590] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11843.323496] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [11843.337877] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [11853.796897] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11853.806663] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11853.838979] Lustre: Skipped 1 previous similar message [11853.858520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11853.864783] Lustre: Skipped 2 previous similar messages [11855.363258] Lustre: server umount lustre-MDT0000 complete [11856.356516] LustreError: 272981:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11856.392813] LustreError: 272981:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [11861.609137] LustreError: 272962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781595111 with bad export cookie 13050389051791041834 [11861.619093] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11861.632586] LustreError: 272962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11862.123483] Lustre: server umount lustre-MDT0001 complete [11878.043706] Lustre: server umount lustre-OST0000 complete [11882.976419] Lustre: 250706:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781595116/real 1781595116] req@ffff8cb888c94700 x1868136958924416/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781595132 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11883.018722] Lustre: 250706:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message [11884.238891] Lustre: server umount lustre-OST0001 complete [11909.085581] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [11923.930541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11924.811404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11924.841794] Lustre: Skipped 3 previous similar messages [11931.924545] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [11944.941294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11945.448459] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [11955.375610] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [11961.310729] Lustre: 279491:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11971.840464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11982.164842] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [11994.900918] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11996.548022] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [11996.588804] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [12007.192458] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [12019.726965] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12024.751449] Lustre: 281372:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [12033.707279] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [12033.776729] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [12046.459469] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 03:34:53 (1781595293) [12063.204267] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12063.214314] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12063.217488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12063.226638] Lustre: Skipped 6 previous similar messages [12063.244322] Lustre: Skipped 3 previous similar messages [12069.549830] Lustre: server umount lustre-MDT0000 complete [12074.785785] LustreError: 278330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781595324 with bad export cookie 13050389051791048638 [12074.816981] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12089.313052] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12089.937475] Lustre: server umount lustre-MDT0001 complete [12110.307428] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12110.504468] Lustre: server umount lustre-OST0000 complete [12115.972976] Lustre: server umount lustre-OST0001 complete [12125.665934] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_hostid [12139.331761] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [12212.855952] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [12229.020488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12229.372642] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12229.457455] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12229.659826] Lustre: lustre-MDT0000: new disk, initializing [12229.824631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12229.832159] Lustre: Skipped 3 previous similar messages [12229.856374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12237.686245] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [12253.158040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12253.378251] Lustre: 285643:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12253.892827] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12261.123225] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [12267.916250] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12277.549091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12277.990456] Lustre: 287280:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12279.981567] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12287.902491] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [12301.711284] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12302.079649] Lustre: 288153:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12304.046629] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12304.061450] Lustre: Skipped 1 previous similar message [12304.159682] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12311.051179] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [12324.793360] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12351.733357] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 03:39:58 (1781595598) [12355.558983] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12355.562991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12355.586404] Lustre: Skipped 5 previous similar messages [12355.601274] Lustre: Skipped 11 previous similar messages [12361.213539] Lustre: server umount lustre-MDT0000 complete [12365.821175] LustreError: 288161:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12365.851360] LustreError: 288161:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 22 previous similar messages [12366.774638] LustreError: 285636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781595616 with bad export cookie 13050389051791056107 [12366.779293] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12366.812621] LustreError: 285636:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [12367.179933] Lustre: server umount lustre-MDT0001 complete [12382.328525] Lustre: server umount lustre-OST0000 complete [12386.280388] Lustre: 250704:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781595620/real 1781595620] req@ffff8cb88cb8a300 x1868136959098240/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781595636 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12388.270743] Lustre: server umount lustre-OST0001 complete [12401.599205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12408.703584] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [12420.620762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12426.864679] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [12437.010596] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12444.043473] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12452.944915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12460.982717] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [12470.838677] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12499.568153] loop1: detected capacity change from 0 to 204800000 [12499.849201] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12499.891177] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12499.942240] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12508.304234] loop1: detected capacity change from 0 to 204800000 [12516.154070] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12516.271652] LustreError: 290392:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) fs25837: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [12516.290875] LustreError: 290392:0:(mgs_llog.c:731:mgs_check_index()) Can't get db for fs25837 [12516.316345] LustreError: 290392:0:(mgs_handler.c:517:mgs_target_reg()) fs25837-MDT0001: Error checking target index: rc = -13 [12516.331487] LustreError: fs25837-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [12516.365917] LustreError: 294124:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -13 [12516.376798] LustreError: 294124:0:(tgt_mount.c:2084:server_put_super()) no obd fs25837-MDT0001 [12516.392114] LustreError: 294124:0:(tgt_mount.c:116:server_deregister_mount()) fs25837-MDT0001 not registered [12516.491275] Lustre: server umount fs25837-MDT0001 complete [12516.494436] LustreError: 294124:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [12528.107946] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12528.295743] Lustre: Setting parameter fs25837-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs25837-MDT0001 [12528.306368] Lustre: Skipped 1 previous similar message [12528.399695] Lustre: srv-fs25837-MDT0001: No data found on store. Initialize space. [12528.409809] Lustre: Skipped 4 previous similar messages [12528.568287] Lustre: fs25837-MDT0001: new disk, initializing [12528.573823] Lustre: Skipped 3 previous similar messages [12528.818109] Lustre: srv-fs25837-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [12528.831372] Lustre: 294593:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs25837-MDT0001-mdtlov: can't create new llog: rc = -115 [12528.856934] LustreError: 294593:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs25837-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [12536.629789] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [12544.494228] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12546.181031] Lustre: server umount fs25837-MDT0001 complete [12565.200533] loop1: detected capacity change from 0 to 204800000 [12572.583666] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12573.087528] Lustre: 296161:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs25837-MDT0001-mdtlov: can't create new llog: rc = -115 [12573.105083] LustreError: 296161:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs25837-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [12578.702305] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [12584.836450] Lustre: server umount fs25837-MDT0001 complete [12600.290359] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12600.301470] LustreError: Skipped 1 previous similar message [12600.311388] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12600.334539] Lustre: Skipped 2 previous similar messages [12602.387178] Lustre: server umount lustre-MDT0000 complete [12613.052062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12613.339529] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12618.774188] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [12619.711645] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [12631.098702] loop1: detected capacity change from 0 to 204800000 [12640.209613] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12641.117793] Lustre: 298610:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs25837-MDT0001-mdtlov: can't create new llog: rc = -115 [12641.131348] LustreError: 298610:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs25837-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [12647.838996] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [12655.532835] Lustre: server umount fs25837-MDT0001 complete [12685.283212] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12685.298377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12685.314456] Lustre: Skipped 9 previous similar messages [12689.014499] Lustre: server umount lustre-OST0000 complete [12707.296762] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12707.619971] Lustre: server umount lustre-MDT0000 complete [12712.317749] LustreError: 290382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781595962 with bad export cookie 13050389051791058788 [12712.323731] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12712.333490] LustreError: 290382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12712.778353] Lustre: server umount lustre-MDT0001 complete [12725.528606] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [12728.487416] Key type lgssc unregistered [12728.798251] LNet: 300956:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12728.811967] LNetError: 300956:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12728.839639] LNet: Removed LNI 192.168.206.149@tcp [12729.799470] Key type .llcrypt unregistered [12729.807471] Key type ._llcrypt unregistered [12755.386541] Key type ._llcrypt registered [12755.388973] Key type .llcrypt registered [12755.541929] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [12757.122882] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12757.164108] alg: No test for adler32 (adler32-zlib) [12758.261642] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [12758.506478] LNet: Added LNI 192.168.206.149@tcp [8/256/0/180] [12760.216211] Key type lgssc registered [12761.555306] Lustre: Echo OBD driver; http://www.lustre.org/ [12771.495488] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12771.512211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12773.448554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12779.350216] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [12783.587786] LustreError: 302232:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12783.627052] LustreError: 302232:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12788.708958] LustreError: 302233:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12790.117256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12790.548480] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12795.749899] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [12804.538496] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12811.770134] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12820.704799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12821.348111] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12829.063419] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [12835.694296] LustreError: 304061:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12837.360783] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12840.931916] LustreError: 304064:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12849.139356] LustreError: 304064:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12849.163512] LustreError: 304064:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [12859.378335] LustreError: 304063:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12859.414229] LustreError: 304063:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [12868.044522] loop1: detected capacity change from 0 to 204800000 [12868.414324] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12868.440167] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12868.464833] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12875.524606] loop1: detected capacity change from 0 to 204800000 [12879.851132] LustreError: 304064:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12879.880242] LustreError: 304064:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [12882.816370] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12883.011854] Lustre: fs11385-OST0000: new disk, initializing [12883.022539] Lustre: srv-fs11385-OST0000: No data found on store. Initialize space. [12883.036838] Lustre: 305991:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12883.189594] Lustre: fs11385-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12890.435676] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [12900.623944] Lustre: server umount fs11385-OST0000 complete [12915.701421] LustreError: 304686:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12915.713585] LustreError: 304686:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 19 previous similar messages [12915.719063] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12915.892902] loop1: detected capacity change from 0 to 204800000 [12920.821771] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12922.319383] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12922.716396] Lustre: fs11385-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12931.696816] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [12946.194659] loop2: detected capacity change from 0 to 204800000 [12946.601601] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12946.618797] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12946.639086] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12953.740609] loop2: detected capacity change from 0 to 204800000 [12961.048635] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12961.179312] LustreError: fs11385-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [12961.189314] LustreError: fs11385-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [12961.196556] LustreError: 309032:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -13 [12961.200781] LustreError: 309032:0:(tgt_mount.c:2084:server_put_super()) no obd fs11385-OST0001 [12961.204040] LustreError: 309032:0:(tgt_mount.c:116:server_deregister_mount()) fs11385-OST0001 not registered [12961.232959] Lustre: server umount fs11385-OST0001 complete [12961.246884] LustreError: 309032:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [12976.933841] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12977.147941] Lustre: fs11385-OST0001: new disk, initializing [12977.151906] Lustre: srv-fs11385-OST0001: No data found on store. Initialize space. [12977.167658] Lustre: 309584:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12977.279739] Lustre: fs11385-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12980.306237] LustreError: 304061:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12980.335540] LustreError: 304061:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 37 previous similar messages [12984.825025] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [12994.632038] Lustre: server umount fs11385-OST0000 complete [13005.215846] Lustre: server umount fs11385-OST0001 complete [13019.488825] loop1: detected capacity change from 0 to 204800000 [13026.487911] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13026.801232] Lustre: fs11385-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13033.943465] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [13042.228995] loop2: detected capacity change from 0 to 204800000 [13050.147698] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13058.784747] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [13069.304288] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [13070.195554] Lustre: server umount fs11385-OST0000 complete [13074.407179] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [13081.096527] Lustre: server umount fs11385-OST0001 complete [13096.942925] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13096.952205] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13096.975924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13100.002119] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13100.026057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13101.922624] Lustre: server umount lustre-OST0000 complete [13108.200671] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13108.204262] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13108.215154] LustreError: Skipped 1 previous similar message [13108.230905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13108.230919] Lustre: Skipped 1 previous similar message [13113.315377] LustreError: 302232:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13113.337673] LustreError: 302232:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 71 previous similar messages [13113.434822] Lustre: server umount lustre-MDT0000 complete [13117.725365] LustreError: 314008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781596367 with bad export cookie 2527665188629397542 [13117.731813] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13117.738286] LustreError: 314008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13118.108465] Lustre: server umount lustre-MDT0001 complete [13132.471254] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [13135.865753] Key type lgssc unregistered [13136.377319] LNet: 315514:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13136.397583] LNetError: 315514:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13136.420621] LNet: Removed LNI 192.168.206.149@tcp [13137.594336] Key type .llcrypt unregistered [13137.596361] Key type ._llcrypt unregistered [13175.540443] Key type ._llcrypt registered [13175.544417] Key type .llcrypt registered [13175.726038] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_hostid [13193.794710] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [13195.552677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13195.656353] alg: No test for adler32 (adler32-zlib) [13196.863394] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [13197.186908] LNet: Added LNI 192.168.206.149@tcp [8/256/0/180] [13198.976251] Key type lgssc registered [13200.456680] Lustre: Echo OBD driver; http://www.lustre.org/ [13243.688385] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13243.731586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13245.339910] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13245.447291] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13245.627523] Lustre: lustre-MDT0000: new disk, initializing [13245.802107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13245.951090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13252.793141] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [13266.809208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13267.021042] Lustre: 319379:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13267.137677] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13267.145348] Lustre: Skipped 1 previous similar message [13267.321369] Lustre: lustre-MDT0001: new disk, initializing [13267.476908] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13267.526896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13267.562868] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13272.769856] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [13283.685673] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13290.182596] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13300.122884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13300.429438] Lustre: lustre-OST0000: new disk, initializing [13300.435328] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13300.448940] Lustre: 321348:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13300.566217] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13308.984615] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [13309.065071] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13309.089357] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13309.276321] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13320.038256] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13322.329458] Lustre: Mounted lustre-client [13323.441637] Lustre: Unmounted lustre-client [13337.582804] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13337.599546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13337.611678] Lustre: Skipped 1 previous similar message [13341.405189] Lustre: server umount lustre-OST0000 complete [13347.812387] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13347.819216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13347.841850] Lustre: Skipped 2 previous similar messages [13347.866112] Lustre: Skipped 1 previous similar message [13352.235272] Lustre: server umount lustre-MDT0000 complete [13356.374600] LustreError: 319372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781596606 with bad export cookie 5811294630078061703 [13356.385525] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13356.389994] LustreError: 319372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13356.804975] Lustre: server umount lustre-MDT0001 complete [13368.162401] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [13371.630990] Key type lgssc unregistered [13380.194084] LNet: 323511:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13380.209173] LNetError: 323511:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13452.906343] LNet: Removed LNI 192.168.206.149@tcp [13454.165556] Key type .llcrypt unregistered [13454.170215] Key type ._llcrypt unregistered [13476.832201] Key type ._llcrypt registered [13476.833868] Key type .llcrypt registered [13477.070744] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_hostid [13499.140633] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [13500.453573] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13500.485114] alg: No test for adler32 (adler32-zlib) [13501.787667] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [13502.024061] LNet: Added LNI 192.168.206.149@tcp [8/256/0/180] [13503.747163] Key type lgssc registered [13505.033607] Lustre: Echo OBD driver; http://www.lustre.org/ [13542.655635] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 03:59:49 (1781596789) [13550.100417] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13550.135332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13551.370739] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13551.444204] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13551.555157] Lustre: lustre-MDT0000: new disk, initializing [13551.665260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13551.694702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13556.296366] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [13568.227859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13568.309487] Lustre: 326738:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13568.372644] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13568.376970] Lustre: Skipped 1 previous similar message [13568.477123] Lustre: lustre-MDT0001: new disk, initializing [13568.592844] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13568.634801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13568.665363] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13573.360540] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [13584.407554] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13591.099770] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13599.206895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13599.606888] Lustre: lustre-OST0000: new disk, initializing [13599.614431] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13599.630590] Lustre: 328707:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13599.785281] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13607.030375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13607.047129] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13607.231694] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13608.091659] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [13621.348316] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13638.116149] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13638.132122] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13638.148743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13643.187797] Lustre: server umount lustre-OST0000 complete [13648.360177] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13648.368093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13648.396600] Lustre: Skipped 2 previous similar messages [13648.432428] Lustre: Skipped 2 previous similar messages [13653.481676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13653.493356] Lustre: Skipped 1 previous similar message [13658.597460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13658.605304] Lustre: Skipped 1 previous similar message [13662.176234] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13664.345267] Lustre: server umount lustre-MDT0000 complete [13668.686992] LustreError: 326731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781596918 with bad export cookie 1093631082196215850 [13668.692329] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13668.698265] LustreError: 326731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13669.163981] Lustre: server umount lustre-MDT0001 complete [13681.275760] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [13684.294528] Key type lgssc unregistered [13684.674988] LNet: 330627:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13684.716290] LNetError: 330627:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13684.754401] LNet: Removed LNI 192.168.206.149@tcp [13685.611932] Key type .llcrypt unregistered [13685.613946] Key type ._llcrypt unregistered [13709.989773] Key type ._llcrypt registered [13709.998354] Key type .llcrypt registered [13710.166636] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [13712.044800] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13712.089807] alg: No test for adler32 (adler32-zlib) [13713.513366] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [13713.806490] LNet: Added LNI 192.168.206.149@tcp [8/256/0/180] [13715.528347] Key type lgssc registered [13716.936870] Lustre: Echo OBD driver; http://www.lustre.org/ [13726.611933] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13726.648453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13728.256803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13733.415224] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [13737.448352] LustreError: 331905:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13737.464087] LustreError: 331905:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13742.562562] LustreError: 331904:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13743.577828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13744.047242] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13750.918123] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [13761.171158] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13767.129794] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13774.825412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13775.105612] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13781.214434] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [13789.334808] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13810.858740] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13812.906285] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13818.340837] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13818.350736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13818.351524] Lustre: Skipped 2 previous similar messages [13818.386635] Lustre: Skipped 2 previous similar messages [13822.078955] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.49@tcp (stopping) [13823.460654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13823.468663] Lustre: Skipped 2 previous similar messages [13823.869568] Lustre: server umount lustre-MDT0000 complete [13827.162137] LustreError: 334847:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13828.011396] LustreError: 331884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781597077 with bad export cookie 17831070327784691747 [13828.027994] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13828.031640] LustreError: 331884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13828.585435] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13828.601369] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13832.266568] LustreError: 333136:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13832.290296] LustreError: 333136:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13833.698103] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13833.704904] Lustre: Skipped 1 previous similar message [13837.384903] LustreError: 331901:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.49@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13837.410115] LustreError: 331901:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13842.403110] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13842.504635] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.49@tcp (stopping) [13842.521462] Lustre: Skipped 2 previous similar messages [13842.759063] Lustre: server umount lustre-MDT0001 complete [13847.848071] Lustre: server umount lustre-OST0000 complete [13858.364591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13863.877031] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [13867.092205] Lustre: 336112:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.149@tcp [13869.164163] Lustre: server umount MGS complete [13881.770807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13882.385192] LustreError: 336686:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13882.399177] LustreError: 336686:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13882.448319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13887.917474] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [13898.754721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13899.131610] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13905.114581] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [13916.136571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13916.525904] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13924.136612] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [13933.341539] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13935.397764] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [13941.765218] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13944.089790] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13949.932383] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [13949.945885] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13949.977324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13953.539178] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13964.256208] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13964.521585] Lustre: server umount lustre-OST0000 complete [13970.916902] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13970.928554] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13975.876302] Lustre: server umount lustre-MDT0000 complete [13979.110129] LustreError: 336681:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13979.133936] LustreError: 336681:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [13980.518458] LustreError: 336667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781597230 with bad export cookie 17831070327784692580 [13980.521319] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13980.537159] LustreError: 336667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13981.007372] Lustre: server umount lustre-MDT0001 complete [13995.531340] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [13999.357816] Key type lgssc unregistered [13999.753220] LNet: 340123:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13999.775413] LNetError: 340123:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13999.825497] LNet: Removed LNI 192.168.206.149@tcp [14001.045554] Key type .llcrypt unregistered [14001.047975] Key type ._llcrypt unregistered [14025.919512] Key type ._llcrypt registered [14025.924798] Key type .llcrypt registered [14026.062456] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_hostid [14046.904421] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [14048.467401] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14048.493836] alg: No test for adler32 (adler32-zlib) [14049.628144] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [14049.875134] LNet: Added LNI 192.168.206.149@tcp [8/256/0/180] [14051.552208] Key type lgssc registered [14053.004225] Lustre: Echo OBD driver; http://www.lustre.org/ [14096.749230] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14096.820402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14098.198727] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14098.265797] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14098.376520] Lustre: lustre-MDT0000: new disk, initializing [14098.540520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14098.590553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14103.996378] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [14116.829043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14117.085660] Lustre: 343290:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14117.181178] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14117.191949] Lustre: Skipped 1 previous similar message [14117.296964] Lustre: lustre-MDT0001: new disk, initializing [14117.492815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14117.541332] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14117.558939] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14122.869513] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [14135.470328] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14142.443758] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14150.787706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14151.246956] Lustre: lustre-OST0000: new disk, initializing [14151.255951] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14151.268376] Lustre: 345263:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14151.419445] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14158.935929] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [14159.956550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14159.969367] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14160.179507] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14169.134886] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14175.049636] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [14175.346803] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [14180.494975] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [14180.687983] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [14185.957818] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14185.960519] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14185.970185] LustreError: Skipped 1 previous similar message [14185.974574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14185.991218] Lustre: Skipped 1 previous similar message [14188.802249] Lustre: server umount lustre-OST0000 complete [14194.145606] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14194.155984] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14194.182458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14194.188682] Lustre: Skipped 1 previous similar message [14196.195575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14196.198349] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14198.929879] Lustre: server umount lustre-MDT0000 complete [14203.061153] LustreError: 343282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781597452 with bad export cookie 14211093009955807014 [14203.063282] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14203.071683] LustreError: 343282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14203.448512] Lustre: server umount lustre-MDT0001 complete [14213.959160] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 04:11:01 (1781597461) [14232.647297] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_hostid [14243.772523] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [14286.563856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14286.774633] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14286.813282] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14286.886845] Lustre: lustre-MDT0000: new disk, initializing [14287.049941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14287.090037] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14293.203605] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [14308.068248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14308.205419] Lustre: 350021:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14308.286345] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14308.289772] Lustre: Skipped 1 previous similar message [14308.395473] Lustre: lustre-MDT0001: new disk, initializing [14308.611426] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14308.662632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14308.684577] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14314.425890] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [14324.072913] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14329.828954] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14336.992618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14337.350279] Lustre: lustre-OST0000: new disk, initializing [14337.357271] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14337.365448] Lustre: 351993:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14337.478480] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14343.297375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14343.315704] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14343.457436] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14344.554607] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [14354.536698] Lustre: DEBUG MARKER: oleg649-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14399.975633] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14399.986054] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14399.998322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14400.006427] Lustre: Skipped 1 previous similar message [14400.016703] Lustre: Skipped 2 previous similar messages [14405.094139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14405.109411] Lustre: Skipped 1 previous similar message [14412.256381] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14412.694431] Lustre: server umount lustre-OST0000 complete [14420.452709] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14420.455031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14420.480541] Lustre: Skipped 1 previous similar message [14420.500142] Lustre: Skipped 3 previous similar messages [14423.860586] Lustre: server umount lustre-MDT0000 complete [14425.570386] LustreError: 352864:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14425.604945] LustreError: 352864:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [14429.480150] LustreError: 350013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781597679 with bad export cookie 14211093009955807889 [14429.497244] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14429.505202] LustreError: 350013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14430.051937] Lustre: server umount lustre-MDT0001 complete [14443.218147] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [14446.931284] Key type lgssc unregistered [14447.312546] LNet: 354170:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14447.338499] LNetError: 354170:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14447.358866] LNet: Removed LNI 192.168.206.149@tcp [14448.303655] Key type .llcrypt unregistered [14448.306457] Key type ._llcrypt unregistered [14555.474566] Key type ._llcrypt registered [14555.479167] Key type .llcrypt registered [14555.637714] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [14556.067393] Key type .llcrypt unregistered [14556.069613] Key type ._llcrypt unregistered [14647.674757] Key type ._llcrypt registered [14647.677318] Key type .llcrypt registered [14647.779984] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [14648.121854] Key type .llcrypt unregistered [14648.125372] Key type ._llcrypt unregistered [14689.462050] Key type ._llcrypt registered [14689.466805] Key type .llcrypt registered [14689.675770] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [14690.077427] Key type .llcrypt unregistered [14690.082571] Key type ._llcrypt unregistered [14725.884512] Key type ._llcrypt registered [14725.886510] Key type .llcrypt registered [14725.971584] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [14726.273555] Key type .llcrypt unregistered [14726.276565] Key type ._llcrypt unregistered [14816.775590] Key type ._llcrypt registered [14816.778686] Key type .llcrypt registered [14816.909730] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [14817.256486] Key type .llcrypt unregistered [14817.264260] Key type ._llcrypt unregistered [14851.727694] Key type ._llcrypt registered [14851.730724] Key type .llcrypt registered [14851.828411] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [14852.109250] Key type .llcrypt unregistered [14852.114523] Key type ._llcrypt unregistered [14883.890212] Key type ._llcrypt registered [14883.894581] Key type .llcrypt registered [14884.047468] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [14884.503474] Key type .llcrypt unregistered [14884.507226] Key type ._llcrypt unregistered [14920.022395] Key type ._llcrypt registered [14920.024982] Key type .llcrypt registered [14920.138224] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [14920.433356] Key type .llcrypt unregistered [14920.445722] Key type ._llcrypt unregistered [14955.900973] Key type ._llcrypt registered [14955.906872] Key type .llcrypt registered [14956.068848] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [14956.404477] Key type .llcrypt unregistered [14956.407516] Key type ._llcrypt unregistered [14992.973695] Key type ._llcrypt registered [14992.980091] Key type .llcrypt registered [14993.229222] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [14993.612398] Key type .llcrypt unregistered [14993.617971] Key type ._llcrypt unregistered [15066.998637] Key type ._llcrypt registered [15067.002933] Key type .llcrypt registered [15067.113805] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [15067.405779] Key type .llcrypt unregistered [15067.412965] Key type ._llcrypt unregistered [15106.976356] Key type ._llcrypt registered [15106.978336] Key type .llcrypt registered [15107.236897] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [15107.828642] Key type .llcrypt unregistered [15107.845969] Key type ._llcrypt unregistered [15145.441947] Key type ._llcrypt registered [15145.444975] Key type .llcrypt registered [15145.553447] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [15145.862492] Key type .llcrypt unregistered [15145.863806] Key type ._llcrypt unregistered [15210.849397] Key type ._llcrypt registered [15210.853478] Key type .llcrypt registered [15211.058172] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_hostid [15231.444221] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [15233.459201] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15233.533201] alg: No test for adler32 (adler32-zlib) [15234.741321] Lustre: Lustre: Build Version: 2.17.54_40_g386d0a3 [15235.119847] LNet: Added LNI 192.168.206.149@tcp [8/256/0/180] [15236.881115] Key type lgssc registered [15238.201807] Lustre: Echo OBD driver; http://www.lustre.org/ [15282.940867] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15282.975977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15284.476358] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [15284.555450] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [15284.718588] Lustre: tst250fs-MDT0000: new disk, initializing [15284.920539] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15284.971171] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15290.704428] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [15305.332121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15305.544358] Lustre: 374521:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15305.722078] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [15305.729136] Lustre: Skipped 1 previous similar message [15305.950163] Lustre: tst250fs-MDT0001: new disk, initializing [15306.116565] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15306.177422] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15306.190239] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15313.222522] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [15320.716669] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [15329.196070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15329.576476] Lustre: tst250fs-OST0000: new disk, initializing [15329.586833] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [15329.601861] Lustre: 376157:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15329.771181] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15330.880974] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15330.900085] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15331.094737] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15337.188281] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [15351.700534] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15351.929378] Lustre: tst250fs-OST0001: new disk, initializing [15351.933667] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [15351.944619] Lustre: 377030:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15352.097482] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [15359.709577] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [15359.722550] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [15359.898816] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [15360.031889] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_default_debug -1 all [15367.713845] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [15370.289486] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [15370.292589] Lustre: Skipped 1 previous similar message [15393.253541] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15393.265972] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15393.281350] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [15395.816138] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [15395.816593] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15395.834071] Lustre: Skipped 1 previous similar message [15395.879989] Lustre: Skipped 2 previous similar messages [15398.593478] Lustre: server umount tst250fs-MDT0000 complete [15404.079069] LustreError: 377029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781598653 with bad export cookie 1008922917859282098 [15404.086185] LustreError: MGC192.168.206.149@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15404.091727] LustreError: 377029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15404.710993] Lustre: server umount tst250fs-MDT0001 complete [15419.818080] Lustre: server umount tst250fs-OST0000 complete [15421.600306] Lustre: 372745:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781598655/real 1781598655] req@ffff8cb9bfb99c00 x1868141412034560/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781598671 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15421.647194] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15424.443386] Lustre: server umount tst250fs-OST0001 complete [15449.664811] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_hostid [15461.700770] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [15503.758212] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 04:32:31 (1781598751) [15505.856914] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [15527.502602] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing set_hostid [15543.217693] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing load_modules_local [15583.144889] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15182 sec ========= 04:33:50 (1781598830) [15585.685466] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 04:33:52 (1781598832) === [15590.353884] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 04:33:57 (1781598837) === [15643.197580] Lustre: DEBUG MARKER: oleg649-server.virtnet: executing unload_modules_local [15647.406633] Key type lgssc unregistered [15647.980112] LNet: 386187:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15647.991106] LNetError: 386187:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15648.015441] LNet: Removed LNI 192.168.206.149@tcp [15649.630233] Key type .llcrypt unregistered [15649.645142] Key type ._llcrypt unregistered