[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 494650330 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003120] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005011] kvm-guest: setup PV IPIs [ 0.008420] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011015] pid_max: default: 32768 minimum: 301 [ 0.013132] LSM: Security Framework initializing [ 0.015076] Yama: becoming mindful. [ 0.016009] SELinux: Initializing. [ 0.017068] *** VALIDATE selinux *** [ 0.024730] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028591] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029169] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031003] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032158] *** VALIDATE tmpfs *** [ 0.034141] *** VALIDATE proc *** [ 0.035349] *** VALIDATE cgroup *** [ 0.036018] *** VALIDATE cgroup2 *** [ 0.038128] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.040010] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.041014] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.042041] Spectre V2 : User space: Vulnerable [ 0.043016] Speculative Store Bypass: Vulnerable [ 0.046992] debug: unmapping init [mem 0xffffffffb0859000-0xffffffffb0860fff] [ 0.050096] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.051764] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.052035] ... version: 2 [ 0.053017] ... bit width: 48 [ 0.054020] ... generic registers: 4 [ 0.055018] ... value mask: 0000ffffffffffff [ 0.056022] ... max period: 00007fffffffffff [ 0.057017] ... fixed-purpose events: 3 [ 0.058016] ... event mask: 000000070000000f [ 0.059280] rcu: Hierarchical SRCU implementation. [ 0.061769] smp: Bringing up secondary CPUs ... [ 0.062616] x86: Booting SMP configuration: [ 0.063028] .... node #0, CPUs: #1 #2 #3 [ 0.066441] smp: Brought up 1 node, 4 CPUs [ 0.068014] smpboot: Max logical packages: 1 [ 0.069018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.152023] node 0 deferred pages initialised in 81ms [ 0.155393] devtmpfs: initialized [ 0.156223] x86/mm: Memory block size: 128MB [ 0.158926] gcov: version magic: 0x41383552 [ 0.159598] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.164124] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.167355] pinctrl core: initialized pinctrl subsystem [ 0.169237] [ 0.169883] ************************************************************* [ 0.173014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.176011] ** ** [ 0.179015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.182013] ** ** [ 0.185013] ** This means that this kernel is built to expose internal ** [ 0.188013] ** IOMMU data structures, which may compromise security on ** [ 0.190017] ** your system. ** [ 0.193014] ** ** [ 0.196012] ** If you see this message and you are not debugging the ** [ 0.199014] ** kernel, report this immediately to your vendor! ** [ 0.201011] ** ** [ 0.204013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.207014] ************************************************************* [ 0.210918] NET: Registered protocol family 16 [ 0.213520] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.216057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.219065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.222105] cpuidle: using governor menu [ 0.223864] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.224743] PCI: Using configuration type 1 for base access [ 0.225122] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.232127] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.235067] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.239041] cryptd: max_cpu_qlen set to 1000 [ 0.242257] ACPI: Added _OSI(Module Device) [ 0.244017] ACPI: Added _OSI(Processor Device) [ 0.246013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.247013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.252023] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.258325] ACPI: Interpreter enabled [ 0.260069] ACPI: PM: (supports S0 S3 S4 S5) [ 0.261011] ACPI: Using IOAPIC for interrupt routing [ 0.263115] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.266608] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.276556] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.279046] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.282021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.286095] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.292701] acpiphp: Slot [2] registered [ 0.294269] acpiphp: Slot [5] registered [ 0.296160] acpiphp: Slot [6] registered [ 0.297121] acpiphp: Slot [7] registered [ 0.299144] acpiphp: Slot [8] registered [ 0.302218] acpiphp: Slot [9] registered [ 0.304128] acpiphp: Slot [10] registered [ 0.305121] acpiphp: Slot [3] registered [ 0.307170] acpiphp: Slot [4] registered [ 0.308168] acpiphp: Slot [11] registered [ 0.310100] acpiphp: Slot [12] registered [ 0.312214] acpiphp: Slot [13] registered [ 0.313161] acpiphp: Slot [14] registered [ 0.315092] acpiphp: Slot [15] registered [ 0.316108] acpiphp: Slot [16] registered [ 0.318093] acpiphp: Slot [17] registered [ 0.320099] acpiphp: Slot [18] registered [ 0.321126] acpiphp: Slot [19] registered [ 0.323161] acpiphp: Slot [20] registered [ 0.324098] acpiphp: Slot [21] registered [ 0.326106] acpiphp: Slot [22] registered [ 0.328182] acpiphp: Slot [23] registered [ 0.329099] acpiphp: Slot [24] registered [ 0.331102] acpiphp: Slot [25] registered [ 0.333136] acpiphp: Slot [26] registered [ 0.334173] acpiphp: Slot [27] registered [ 0.336098] acpiphp: Slot [28] registered [ 0.338104] acpiphp: Slot [29] registered [ 0.339100] acpiphp: Slot [30] registered [ 0.341103] acpiphp: Slot [31] registered [ 0.342094] PCI host bridge to bus 0000:00 [ 0.344023] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.347024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.349024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.351036] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.353026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.357028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.359190] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.362280] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.365363] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.375740] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.380629] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.383084] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.386018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.388013] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.390291] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.392702] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.395044] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.397810] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.403016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.418014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.423019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.431014] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.443017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.451015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.473016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.485204] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.495015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.501014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.520014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.531894] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.539017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.547015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.565017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.577531] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.585013] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.595020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.613025] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.624241] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.631016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.640015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.658020] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.669198] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.676020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.687019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.705026] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.717887] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.721310] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.723364] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.726377] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.728240] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.736051] iommu: Default domain type: Passthrough [ 0.737516] SCSI subsystem initialized [ 0.739161] ACPI: bus type USB registered [ 0.740192] usbcore: registered new interface driver usbfs [ 0.742116] usbcore: registered new interface driver hub [ 0.745131] usbcore: registered new device driver usb [ 0.747183] pps_core: LinuxPPS API ver. 1 registered [ 0.749025] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.754105] PTP clock support registered [ 0.757044] EDAC MC: Ver: 3.0.0 [ 0.759144] PCI: Using ACPI for IRQ routing [ 0.763009] NetLabel: Initializing [ 0.764008] NetLabel: domain hash size = 128 [ 0.766013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.769070] NetLabel: unlabeled traffic allowed by default [ 0.772158] vgaarb: loaded [ 0.775232] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.777015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.782414] clocksource: Switched to clocksource kvm-clock [ 0.894535] VFS: Disk quotas dquot_6.6.0 [ 0.896265] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.899331] *** VALIDATE ramfs *** [ 0.900697] *** VALIDATE hugetlbfs *** [ 0.902474] pnp: PnP ACPI init [ 0.904746] pnp: PnP ACPI: found 6 devices [ 0.922337] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.926218] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.928784] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.931375] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.934292] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.937077] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.940160] NET: Registered protocol family 2 [ 0.942647] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.946965] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.950806] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.956694] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.960792] TCP: Hash tables configured (established 65536 bind 65536) [ 0.963724] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.968647] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.971949] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.975078] NET: Registered protocol family 1 [ 0.978123] RPC: Registered named UNIX socket transport module. [ 0.980496] RPC: Registered udp transport module. [ 0.982289] RPC: Registered tcp transport module. [ 0.983685] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.986081] NET: Registered protocol family 44 [ 0.988431] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.990722] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.993255] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.995597] PCI: CLS 0 bytes, default 64 [ 0.997355] Unpacking initramfs... [ 2.432786] debug: unmapping init [mem 0xffff96f1bcc54000-0xffff96f1bffbffff] [ 2.439758] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.443289] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.448154] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.931242] Initialise system trusted keyrings [ 2.933333] Key type blacklist registered [ 2.935626] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.945667] zbud: loaded [ 2.948988] *** VALIDATE nfs *** [ 2.950564] *** VALIDATE nfs4 *** [ 2.952576] pstore: using deflate compression [ 2.958268] Platform Keyring initialized [ 3.079195] NET: Registered protocol family 38 [ 3.081269] Key type asymmetric registered [ 3.083016] Asymmetric key parser 'x509' registered [ 3.084996] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.088326] io scheduler mq-deadline registered [ 3.090991] io scheduler kyber registered [ 3.092618] io scheduler bfq registered [ 3.094542] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.097918] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.101636] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.105203] ACPI: Power Button [PWRF] [ 3.111983] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.119850] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.139179] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.146234] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.164256] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.191799] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.220289] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.224331] Non-volatile memory driver v1.3 [ 3.226312] Linux agpgart interface v0.103 [ 3.257513] virtio_blk virtio1: [vda] 146144 512-byte logical blocks (74.8 MB/71.4 MiB) [ 3.260788] vda: detected capacity change from 0 to 74825728 [ 3.279988] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.284996] vdb: detected capacity change from 0 to 1073741824 [ 3.303194] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.307228] vdc: detected capacity change from 0 to 2621440000 [ 3.322065] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.325461] vdd: detected capacity change from 0 to 2621440000 [ 3.341022] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.344852] vde: detected capacity change from 0 to 4294967296 [ 3.359854] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.363225] vdf: detected capacity change from 0 to 4294967296 [ 3.371624] libphy: Fixed MDIO Bus: probed [ 3.377721] usbcore: registered new interface driver usbserial_generic [ 3.383624] usbserial: USB Serial support registered for generic [ 3.387350] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.391780] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.393678] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.396440] mousedev: PS/2 mouse device common for all mice [ 3.399937] rtc_cmos 00:05: RTC can wake from S4 [ 3.403672] rtc_cmos 00:05: registered as rtc0 [ 3.404895] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.405339] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.411743] intel_pstate: CPU model not supported [ 3.415277] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.419315] hid: raw HID events driver (C) Jiri Kosina [ 3.420728] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.422093] usbcore: registered new interface driver usbhid [ 3.426342] usbhid: USB HID core driver [ 3.428208] drop_monitor: Initializing network drop monitor service [ 3.430189] Initializing XFRM netlink socket [ 3.432547] NET: Registered protocol family 10 [ 3.435343] Segment Routing with IPv6 [ 3.436742] NET: Registered protocol family 17 [ 3.438310] mpls_gso: MPLS GSO support [ 3.443216] RAS: Correctable Errors collector initialized. [ 3.445516] AVX version of gcm_enc/dec engaged. [ 3.447037] AES CTR mode by8 optimization enabled [ 3.524284] sched_clock: Marking stable (3524263581, 0)->(4427170986, -902907405) [ 3.527864] registered taskstats version 1 [ 3.529560] Loading compiled-in X.509 certificates [ 3.531404] zswap: loaded using pool lzo/zbud [ 3.558050] Key type big_key registered [ 3.571832] Key type encrypted registered [ 3.574068] ima: No TPM chip found, activating TPM-bypass! [ 3.576287] ima: Allocated hash algorithm: sha1 [ 3.577714] ima: No architecture policies found [ 3.579651] evm: Initialising EVM extended attributes: [ 3.582064] evm: security.selinux [ 3.583549] evm: security.ima [ 3.584843] evm: security.capability [ 3.586398] evm: HMAC attrs: 0x1 [ 3.589309] rtc_cmos 00:05: setting system clock to 2026-08-25 22:32:19 UTC (1787697139) [ 3.596226] debug: unmapping init [mem 0xffffffffb1803000-0xffffffffb19fffff] [ 3.598574] debug: unmapping init [mem 0xffffffffb0582000-0xffffffffb0858fff] [ 3.608270] Write protecting the kernel read-only data: 28672k [ 3.611903] debug: unmapping init [mem 0xffffffffaec03000-0xffffffffaedfffff] [ 3.616088] debug: unmapping init [mem 0xffffffffaf514000-0xffffffffaf5fffff] [ 3.649227] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.658633] systemd[1]: Detected virtualization kvm. [ 3.660844] systemd[1]: Detected architecture x86-64. [ 3.662833] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.688181] systemd[1]: No hostname configured. [ 3.690135] systemd[1]: Set hostname to . [ 3.692534] random: systemd: uninitialized urandom read (16 bytes read) [ 3.695397] systemd[1]: Initializing machine ID from random generator. [ 3.827222] random: systemd: uninitialized urandom read (16 bytes read) [ 3.830142] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.836396] random: systemd: uninitialized urandom read (16 bytes read) [ 3.844108] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ 3.853141] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. Starting Setup Virtual Console... [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.426434] device-mapper: uevent: version 1.0.3 [ 4.428958] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.194922] virtio_net virtio0 ens2: renamed from eth0 [ 5.223754] random: fast init done [ 5.231918] scsi host0: ata_piix [ 5.242607] scsi host1: ata_piix [ 5.246806] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.250623] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.677394] dracut-initqueue[584]: RTNETLINK answers: File exists [ 10.091488] random: crng init done [ 10.093030] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.515722] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.653641] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.930792] SELinux: Disabled at runtime. [ 11.990860] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.000809] systemd[1]: Detected virtualization kvm. [ 12.003737] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.573631] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.577748] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.584579] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.590277] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.595889] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.608186] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.622702] systemd[1]: Starting Apply Kernel Variables... Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. [ OK ] Reached target rpc_pipefs.target. Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ 12.738985] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.159091] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.528339] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.529888] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.671803] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.695062] EDAC sbridge: Ver: 1.1.2 [ 15.533808] Key type dns_resolver registered [ 15.859821] NFS: Registering the id_resolver key type [ 15.862338] Key type id_resolver registered [ 15.864425] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started RPC Bind. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Daily Cleanup of Temporary Directories. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg344-server login: [ 52.651114] libcfs: loading out-of-tree module taints kernel. [ 52.738900] Key type ._llcrypt registered [ 52.744032] Key type .llcrypt registered [ 52.868781] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 74.890311] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 76.874083] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 76.893440] alg: No test for adler32 (adler32-zlib) [ 78.529771] Lustre: Lustre: Build Version: 2.17.57_80_g4b309f5 [ 79.528598] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 81.257462] Key type lgssc registered [ 83.065709] Lustre: Echo OBD driver; http://www.lustre.org/ [ 83.979062] hrtimer: interrupt took 7249733 ns [ 104.253729] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 147.740824] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 164.728751] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 164.791278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 166.057853] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 166.113489] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 166.211633] Lustre: lustre-MDT0000: new disk, initializing [ 166.324895] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 166.360953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 172.553928] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 190.499760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 190.584333] Lustre: 6523:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 190.611493] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 190.615345] Lustre: Skipped 1 previous similar message [ 190.670808] Lustre: lustre-MDT0001: new disk, initializing [ 190.722986] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 190.763330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 190.775952] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 196.187816] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 203.677507] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 213.697944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 213.875762] Lustre: lustre-OST0000: new disk, initializing [ 213.887842] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 213.926726] Lustre: 8464:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 213.996858] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 214.757521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 214.779329] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 214.883742] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 220.144638] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 234.431726] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 234.614541] Lustre: lustre-OST0001: new disk, initializing [ 234.619274] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 234.625899] Lustre: 9537:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 234.740602] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 241.534184] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 244.280176] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 244.296200] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 244.359711] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 255.266569] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 264.814368] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 273.673416] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing check_logdir /tmp/testlogs/ [ 279.144709] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing yml_node [ 284.288572] Lustre: DEBUG MARKER: Client: 2.17.57.80 [ 286.981766] Lustre: DEBUG MARKER: MDS: 2.17.57.80 [ 289.508829] Lustre: DEBUG MARKER: OSS: 2.17.57.80 [ 291.059628] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Aug 25 18:37:04 EDT 2026 [ 311.324707] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 313.423564] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 331.233262] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 331.239055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 331.265351] Lustre: Skipped 3 previous similar messages [ 331.289637] Lustre: Skipped 3 previous similar messages [ 336.358083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 336.381166] Lustre: Skipped 3 previous similar messages [ 336.919358] Lustre: server umount lustre-MDT0000 complete [ 340.931821] LustreError: 6515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787697476 with bad export cookie 14458728722427868613 [ 340.940560] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 340.943793] LustreError: 6515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 341.473382] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 341.514701] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 341.533669] Lustre: Skipped 1 previous similar message [ 345.505763] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 345.526092] Lustre: Skipped 1 previous similar message [ 350.625156] LustreError: 6532:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 350.625268] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 350.657352] LustreError: 6532:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 350.667490] Lustre: Skipped 1 previous similar message [ 355.296243] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 355.559052] Lustre: server umount lustre-MDT0001 complete [ 361.606267] Lustre: server umount lustre-OST0000 complete [ 366.911451] Lustre: server umount lustre-OST0001 complete [ 374.848458] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 386.875134] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 424.063247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 424.271273] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 424.298633] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 424.371277] Lustre: lustre-MDT0000: new disk, initializing [ 424.439369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 424.464719] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 429.487737] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 440.761483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 440.899781] Lustre: 15090:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 440.992469] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 441.001738] Lustre: Skipped 1 previous similar message [ 441.102844] Lustre: lustre-MDT0001: new disk, initializing [ 441.207445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 441.245961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 441.255567] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 445.583519] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 456.285740] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 461.508661] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 467.900164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 468.242720] Lustre: lustre-OST0000: new disk, initializing [ 468.247388] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 468.267813] Lustre: 17058:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 468.383921] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 470.160234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 470.172960] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 470.318354] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 474.726767] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 483.691927] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 488.457031] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 488.681299] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 493.293871] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 493.479231] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 496.102215] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 496.107198] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 496.111483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 496.115824] Lustre: Skipped 2 previous similar messages [ 496.146885] LustreError: Skipped 1 previous similar message [ 501.688419] Lustre: server umount lustre-OST0000 complete [ 506.340478] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 506.353035] Lustre: Skipped 1 previous similar message [ 512.098231] Lustre: server umount lustre-MDT0000 complete [ 516.345737] LustreError: 15084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787697652 with bad export cookie 14458728722427870146 [ 516.348702] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 516.356188] LustreError: 15084:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 517.265974] Lustre: server umount lustre-MDT0001 complete [ 523.930122] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 18:40:57 (1787697657) [ 546.504766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 547.098729] LustreError: 19933:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 547.119779] LustreError: 19933:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 547.216339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 552.105610] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 552.420150] LustreError: 19934:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 557.556753] LustreError: 19933:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 562.349468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 562.659766] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 562.673598] Lustre: Skipped 7 previous similar messages [ 568.068857] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 576.208669] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 582.196378] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 584.158658] Lustre: Failing over lustre-MDT0000 [ 584.451058] Lustre: server umount lustre-MDT0000 complete [ 587.250141] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 587.260252] LustreError: 19933:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 587.278984] Lustre: Skipped 1 previous similar message [ 597.482518] LustreError: 20662:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 597.514326] LustreError: 20662:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 603.475359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 603.616784] Lustre: 3655:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787697723/real 1787697723] req@ffff96f101f50000 x1874536395080704/t0(0) o400->MGC192.168.203.144@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787697739 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 603.649450] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 603.990700] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 608.039769] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 608.040527] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 608.069404] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 608.806736] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 616.873734] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 623.587248] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 623.587921] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 623.592125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 623.599832] Lustre: Skipped 1 previous similar message [ 627.044625] Lustre: server umount lustre-MDT0000 complete [ 628.721567] LustreError: 19933:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 628.753204] LustreError: 19933:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 631.106258] LustreError: 19913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787697767 with bad export cookie 14458728722427871560 [ 631.119428] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 631.121183] LustreError: 19913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 631.545275] Lustre: server umount lustre-MDT0001 complete [ 641.929405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 642.391868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 642.399156] Lustre: Skipped 2 previous similar messages [ 647.149968] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 656.805232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 663.239239] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 672.709676] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 678.901615] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 681.376228] Lustre: Failing over lustre-MDT0000 [ 681.612269] Lustre: server umount lustre-MDT0000 complete [ 682.409291] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 682.426372] LustreError: 23269:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 682.429789] Lustre: Skipped 1 previous similar message [ 682.459643] LustreError: 23269:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 698.596436] Lustre: 3653:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787697818/real 1787697818] req@ffff96f103b2d180 x1874536395108992/t0(0) o400->MGC192.168.203.144@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787697834 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 698.626344] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 702.230800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 708.903572] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xc8a7b92adc0d04f2 [ 708.922349] Lustre: MGC192.168.203.144@tcp: Connection restored to 0@lo (at 0@lo) [ 708.934858] Lustre: Skipped 1 previous similar message [ 709.335747] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 714.721693] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 714.744586] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 714.799269] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 715.367783] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 726.226808] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 733.154300] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 733.159050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 733.181381] Lustre: Skipped 1 previous similar message [ 733.209480] Lustre: Skipped 2 previous similar messages [ 737.587100] Lustre: server umount lustre-MDT0000 complete [ 742.053396] LustreError: 23250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787697877 with bad export cookie 14458728722427872498 [ 742.070469] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 742.719942] Lustre: server umount lustre-MDT0001 complete [ 752.919475] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 18:44:45 (1787697885) [ 771.644438] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 782.080761] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 822.138340] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 834.244973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 834.497755] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 834.534728] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 834.632808] Lustre: lustre-MDT0000: new disk, initializing [ 834.756673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 834.767145] Lustre: Skipped 2 previous similar messages [ 834.811380] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 840.781347] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 852.581865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 852.733658] Lustre: 29649:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 853.188903] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 858.054871] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 868.649703] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 875.419401] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 884.806675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 885.148826] Lustre: 31617:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 886.480665] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 886.526359] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 886.628858] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 886.672681] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 892.148433] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 902.500690] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 908.749379] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 932.834658] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 932.842413] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 932.856406] LustreError: Skipped 1 previous similar message [ 932.858049] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 932.887180] Lustre: Skipped 1 previous similar message [ 936.795229] Lustre: server umount lustre-OST0000 complete [ 946.688283] Lustre: server umount lustre-MDT0000 complete [ 948.202527] LustreError: 29656:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 948.225247] LustreError: 29656:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [ 950.421262] LustreError: 29640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787698086 with bad export cookie 14458728722427872911 [ 950.426919] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 950.429366] LustreError: 29640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 950.883183] Lustre: server umount lustre-MDT0001 complete [ 962.372239] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 965.909173] Key type lgssc unregistered [ 966.280037] LNet: 33599:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 966.289149] LNetError: 33599:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 966.325241] LNet: Removed LNI 192.168.203.144@tcp [ 967.659150] Key type .llcrypt unregistered [ 967.666372] Key type ._llcrypt unregistered [ 996.504047] Key type ._llcrypt registered [ 996.505594] Key type .llcrypt registered [ 996.648173] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 998.845433] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 998.875713] alg: No test for adler32 (adler32-zlib) [ 1000.173750] Lustre: Lustre: Build Version: 2.17.57_80_g4b309f5 [ 1000.541583] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 1002.320216] Key type lgssc registered [ 1003.570936] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1011.370200] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1011.404686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1012.885318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1017.105135] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1022.448793] LustreError: 35011:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1022.464640] LustreError: 35011:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1026.332208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1026.755277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1031.193549] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1039.736399] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1046.214929] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1054.452917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1054.813826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1062.482614] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1063.413708] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1071.040727] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1071.231607] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1089.778790] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 18:50:22 (1787698222) [ 1093.327546] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1096.929647] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1108.602283] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 18:50:40 (1787698240) [ 1121.735737] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 18:50:54 (1787698254) [ 1123.239641] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1132.669333] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1132.677403] Lustre: Skipped 2 previous similar messages [ 1134.573938] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 18:51:07 (1787698267) [ 1136.785976] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1136.790724] Lustre: Skipped 2 previous similar messages [ 1145.347392] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1145.356447] Lustre: Skipped 4 previous similar messages [ 1162.474550] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 18:51:35 (1787698295) [ 1165.877946] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1170.877513] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1180.593439] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 18:51:53 (1787698313) [ 1183.058241] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1183.064264] Lustre: Skipped 6 previous similar messages [ 1198.253478] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 18:52:10 (1787698330) [ 1214.948133] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1214.949987] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1214.962085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1218.432463] Lustre: server umount lustre-OST0000 complete [ 1225.185119] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1225.185989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1225.202331] Lustre: Skipped 2 previous similar messages [ 1225.216434] Lustre: Skipped 1 previous similar message [ 1230.305502] LustreError: 35741:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1230.320932] LustreError: 35741:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1230.621699] Lustre: server umount lustre-MDT0000 complete [ 1234.355914] LustreError: 34993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787698370 with bad export cookie 9838425980599628599 [ 1234.358748] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1234.367312] LustreError: 34993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1234.914879] Lustre: server umount lustre-MDT0001 complete [ 1245.551911] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 1248.429555] Key type lgssc unregistered [ 1248.701042] LNet: 42876:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1248.715049] LNetError: 42876:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1248.740807] LNet: Removed LNI 192.168.203.144@tcp [ 1249.657397] Key type .llcrypt unregistered [ 1249.660055] Key type ._llcrypt unregistered [ 1266.958146] Key type ._llcrypt registered [ 1266.962411] Key type .llcrypt registered [ 1267.110467] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 1284.018356] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 1285.400180] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1285.429775] alg: No test for adler32 (adler32-zlib) [ 1286.541187] Lustre: Lustre: Build Version: 2.17.57_80_g4b309f5 [ 1286.895764] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 1288.632418] Key type lgssc registered [ 1289.834367] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1326.752536] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1326.793835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1328.207133] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1328.272258] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1328.446312] Lustre: lustre-MDT0000: new disk, initializing [ 1328.572021] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1328.597135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1333.578826] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1343.328989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1343.434911] Lustre: 46033:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1343.492754] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1343.499806] Lustre: Skipped 1 previous similar message [ 1343.602306] Lustre: lustre-MDT0001: new disk, initializing [ 1343.691367] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1343.725439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1343.737522] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1348.054855] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1358.156771] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1363.459991] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1370.250576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1370.547320] Lustre: lustre-OST0000: new disk, initializing [ 1370.554480] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1370.562740] Lustre: 48003:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1370.681199] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1377.231772] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1379.396204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1379.409181] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1379.547921] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1389.587343] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1396.195210] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1396.391201] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1401.579857] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1401.818818] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1405.416804] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1405.418894] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1405.444483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1410.602893] Lustre: server umount lustre-OST0000 complete [ 1415.654698] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1415.663783] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1415.673199] Lustre: Skipped 1 previous similar message [ 1415.674181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1415.674186] Lustre: Skipped 1 previous similar message [ 1415.723744] LustreError: Skipped 1 previous similar message [ 1420.770721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1420.779406] Lustre: Skipped 2 previous similar messages [ 1420.910524] Lustre: server umount lustre-MDT0000 complete [ 1424.658197] LustreError: 46025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787698560 with bad export cookie 10762849509773501780 [ 1424.660819] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1424.666966] LustreError: 46025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1424.985949] Lustre: server umount lustre-MDT0001 complete [ 1435.996191] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 18:56:08 (1787698568) [ 1445.472400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1446.317754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1451.498347] LustreError: 50467:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1451.535636] LustreError: 50467:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1451.993343] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1456.624581] LustreError: 50466:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1461.755128] LustreError: 50467:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1461.802919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1462.269541] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1467.216239] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1475.422332] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1482.436168] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1489.725370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1490.035693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1496.544214] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1504.002080] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1511.759533] LustreError: 52994:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1512.618888] Lustre: Modifying parameter general.timeout=1 in log params [ 1514.335486] Lustre: Modifying parameter general.timeout=3 in log params [ 1514.349136] Lustre: Skipped 1 previous similar message [ 1516.783464] Lustre: Modifying parameter general.timeout=5 in log params [ 1516.791327] Lustre: Skipped 1 previous similar message [ 1521.656487] Lustre: Modifying parameter general.timeout=9 in log params [ 1521.659594] Lustre: Skipped 3 previous similar messages [ 1529.947845] Lustre: Modifying parameter general.timeout=18 in log params [ 1529.951049] Lustre: Skipped 8 previous similar messages [ 1546.774610] Lustre: Modifying parameter general.timeout=34 in log params [ 1546.778450] Lustre: Skipped 15 previous similar messages [ 1579.677734] Lustre: Modifying parameter general.timeout=71 in log params [ 1579.683186] Lustre: Skipped 36 previous similar messages [ 1646.356113] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 18:59:39 (1787698779) [ 1647.492164] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_policies=tbf in log params [ 1647.495465] Lustre: Skipped 59 previous similar messages [ 1649.071450] Lustre: 59941:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1650.686390] Lustre: 60053:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1654.227210] Lustre: 60250:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1654.234888] Lustre: 60250:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1682.891911] Lustre: 61822:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1682.900193] Lustre: 61822:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1682.925078] LustreError: 61822:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1687.000114] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 19:00:20 (1787698820) [ 1700.328449] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1700.333338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1700.353045] Lustre: Skipped 3 previous similar messages [ 1700.375138] Lustre: Skipped 2 previous similar messages [ 1704.909480] Lustre: server umount lustre-MDT0000 complete [ 1705.446408] LustreError: 50466:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1705.493837] LustreError: 50466:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1709.196357] LustreError: 50446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787698845 with bad export cookie 10762849509773502662 [ 1709.209914] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1709.220530] LustreError: 50446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1709.946945] Lustre: server umount lustre-MDT0001 complete [ 1725.479486] Lustre: server umount lustre-OST0000 complete [ 1754.271792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1754.636974] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1754.931989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1759.705274] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1769.081722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1769.234247] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1769.310557] Lustre: 64365:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1775.640084] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1783.262671] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1789.157710] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1796.951457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1797.224605] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1797.620119] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1797.634827] Lustre: Skipped 1 previous similar message [ 1803.392993] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1810.332056] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1811.975367] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1811.988704] Lustre: Skipped 22 previous similar messages [ 1812.054634] Lustre: 66933:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1820.131468] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1820.139354] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1820.157993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1824.553456] Lustre: server umount lustre-OST0000 complete [ 1830.884866] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1830.892483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1830.899854] Lustre: Skipped 2 previous similar messages [ 1830.904676] Lustre: Skipped 2 previous similar messages [ 1834.737515] Lustre: server umount lustre-MDT0000 complete [ 1836.011147] LustreError: 65824:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1836.028917] LustreError: 65824:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1837.904240] LustreError: 65107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787698973 with bad export cookie 10762849509773505763 [ 1837.907557] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1837.913962] LustreError: 65107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1838.282032] Lustre: server umount lustre-MDT0001 complete [ 1849.868734] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 1853.024928] Key type lgssc unregistered [ 1853.323532] LNet: 68146:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1853.335516] LNetError: 68146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1853.348606] LNet: Removed LNI 192.168.203.144@tcp [ 1854.259662] Key type .llcrypt unregistered [ 1854.262364] Key type ._llcrypt unregistered [ 1880.132688] Key type ._llcrypt registered [ 1880.135133] Key type .llcrypt registered [ 1880.306651] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 1881.552218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1881.585667] alg: No test for adler32 (adler32-zlib) [ 1882.656908] Lustre: Lustre: Build Version: 2.17.57_80_g4b309f5 [ 1882.915417] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 1884.624172] Key type lgssc registered [ 1885.818409] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1893.905151] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1893.921073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1895.323374] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1899.574955] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1905.635277] LustreError: 69562:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1905.652944] LustreError: 69562:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1907.748986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1908.324611] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1913.027167] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1921.004603] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1927.185495] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1934.545253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1935.296918] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1942.046275] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1950.272795] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1958.893382] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1969.641079] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1969.657924] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1969.668626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1973.230746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1973.231813] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1973.242682] Lustre: Skipped 1 previous similar message [ 1973.254865] Lustre: Skipped 1 previous similar message [ 1975.396234] Lustre: server umount lustre-MDT0000 complete [ 1978.347420] LustreError: 69563:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1978.378464] LustreError: 69563:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1979.078035] LustreError: 69545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787699114 with bad export cookie 8745167085362528553 [ 1979.093152] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1979.096331] LustreError: 69545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1979.492858] Lustre: server umount lustre-MDT0001 complete [ 1993.573660] Lustre: server umount lustre-OST0000 complete [ 2023.779313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2024.009384] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2024.047045] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2024.363120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2028.980578] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2039.035141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2039.116894] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2039.142347] Lustre: 74432:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2039.599972] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2043.702590] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2052.232182] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2059.392889] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2067.639341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2067.875539] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2068.290745] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2075.829861] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2082.909823] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2084.367738] Lustre: 77010:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2086.219173] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2096.100240] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2096.105127] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2096.112352] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2096.115021] Lustre: Skipped 1 previous similar message [ 2098.815734] Lustre: server umount lustre-OST0000 complete [ 2106.358449] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2106.371868] LustreError: Skipped 1 previous similar message [ 2106.376167] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2106.393771] Lustre: Skipped 1 previous similar message [ 2106.399521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2106.411849] Lustre: Skipped 1 previous similar message [ 2109.707287] Lustre: server umount lustre-MDT0000 complete [ 2111.978714] LustreError: 75385:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2112.005214] LustreError: 75385:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2115.153888] LustreError: 74424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787699251 with bad export cookie 8745167085362529400 [ 2115.157136] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2115.169973] LustreError: 74424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2115.676100] Lustre: server umount lustre-MDT0001 complete [ 2129.789300] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 2132.276284] Key type lgssc unregistered [ 2132.528544] LNet: 78223:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2132.534861] LNetError: 78223:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2132.541870] LNet: Removed LNI 192.168.203.144@tcp [ 2133.058127] Key type .llcrypt unregistered [ 2133.060138] Key type ._llcrypt unregistered [ 2157.526010] Key type ._llcrypt registered [ 2157.527713] Key type .llcrypt registered [ 2157.597756] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 2158.307558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2158.362147] alg: No test for adler32 (adler32-zlib) [ 2159.493612] Lustre: Lustre: Build Version: 2.17.57_80_g4b309f5 [ 2159.681076] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 2161.336176] Key type lgssc registered [ 2162.062985] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2169.623148] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2169.640569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2171.206424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2175.220836] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2181.601645] LustreError: 79636:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2181.612138] LustreError: 79636:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2185.628778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2186.207300] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2191.290289] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2198.743925] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2206.441271] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2213.343882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2213.858706] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2221.197642] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2230.419888] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2237.144298] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2238.386653] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2239.472401] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2242.244090] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2242.247301] Lustre: Skipped 2 previous similar messages [ 2246.655595] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2246.660121] Lustre: Skipped 4 previous similar messages [ 2255.071366] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2255.080948] Lustre: Skipped 7 previous similar messages [ 2271.150417] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2271.158170] Lustre: Skipped 14 previous similar messages [ 2303.473413] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2303.495617] Lustre: Skipped 29 previous similar messages [ 2349.551084] Lustre: 87753:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2352.233561] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 19:11:25 (1787699485) [ 2353.771281] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2355.427362] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 19:11:28 (1787699488) [ 2360.292098] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2360.306844] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2360.334241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2361.317792] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2361.324345] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2366.497150] Lustre: server umount lustre-OST0000 complete [ 2370.530847] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2370.548398] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2370.563199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2370.571512] Lustre: Skipped 1 previous similar message [ 2376.612479] Lustre: server umount lustre-MDT0000 complete [ 2376.685477] LustreError: 80628:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2376.706103] LustreError: 80628:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2380.316862] LustreError: 79615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787699516 with bad export cookie 7264971899393307001 [ 2380.328454] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2380.332019] LustreError: 79615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2380.642288] Lustre: server umount lustre-MDT0001 complete [ 2392.925792] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 2395.888204] Key type lgssc unregistered [ 2396.218157] LNet: 89005:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2396.230780] LNetError: 89005:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2397.291232] LNet: Removed LNI 192.168.203.144@tcp [ 2398.448501] Key type .llcrypt unregistered [ 2398.450954] Key type ._llcrypt unregistered [ 2404.438872] Key type ._llcrypt registered [ 2404.445965] Key type .llcrypt registered [ 2404.649799] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2410.121835] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules [ 2411.946672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2412.003122] alg: No test for adler32 (adler32-zlib) [ 2413.244421] Lustre: Lustre: Build Version: 2.17.57_80_g4b309f5 [ 2413.625392] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 2415.369297] Key type lgssc registered [ 2416.733831] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2418.098793] LustreError: 90256:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2436.516854] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 2440.723809] LustreError: 90256:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2446.180816] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2446.196812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2447.674701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2452.135740] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2458.084187] LustreError: 91279:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2458.119064] LustreError: 91279:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2461.743501] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 19:13:14 (1787699594) [ 2463.211635] LustreError: 91280:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2466.981467] LustreError: 91312:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2466.993194] LustreError: 91312:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2467.008925] LustreError: 91312:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 2467.437496] Lustre: server umount lustre-MDT0000 complete [ 2481.133734] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 2483.625773] Key type lgssc unregistered [ 2483.840810] LNet: 92808:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2483.858192] LNetError: 92808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2483.886262] LNet: Removed LNI 192.168.203.144@tcp [ 2484.593816] Key type .llcrypt unregistered [ 2484.598057] Key type ._llcrypt unregistered [ 2502.811354] Key type ._llcrypt registered [ 2502.813505] Key type .llcrypt registered [ 2502.907248] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 2503.961169] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2504.141721] alg: No test for adler32 (adler32-zlib) [ 2505.305671] Lustre: Lustre: Build Version: 2.17.57_80_g4b309f5 [ 2505.671894] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 2507.362159] Key type lgssc registered [ 2508.681562] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2516.587211] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2516.620235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2518.197229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2523.295322] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2528.744977] LustreError: 94037:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2528.778149] LustreError: 94037:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2532.206644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2533.006161] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2538.251140] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2547.091696] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2553.820301] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2561.224075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2561.988310] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2568.569730] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2576.026192] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2623.474517] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 19:15:56 (1787699756) [ 2625.285900] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2627.206167] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 19:16:00 (1787699760) [ 2633.699827] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2633.708625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2633.722781] Lustre: Skipped 2 previous similar messages [ 2633.732160] Lustre: Skipped 1 previous similar message [ 2637.490910] Lustre: server umount lustre-MDT0000 complete [ 2638.820721] LustreError: 95993:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2638.839427] LustreError: 95993:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2640.894259] LustreError: 94016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787699776 with bad export cookie 7281191961309476589 [ 2640.897121] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2640.904282] LustreError: 94016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2641.220246] Lustre: server umount lustre-MDT0001 complete [ 2655.076651] Lustre: server umount lustre-OST0000 complete [ 2668.617272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2669.042260] LustreError: 98313:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2669.256575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2674.635769] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2674.666187] LustreError: 98314:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2678.795531] LustreError: 98313:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2683.904674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2684.543406] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2689.001702] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2697.016021] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2703.815758] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2719.081802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2719.324446] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2719.335233] LustreError: 98301:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2719.371675] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2719.382448] LustreError: 100398:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2719.389703] LustreError: 100398:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2719.395045] LustreError: 100398:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2719.484466] Lustre: server umount lustre-OST0000 complete [ 2719.487644] LustreError: 100398:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2727.142874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2727.283315] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2727.295644] LustreError: 98301:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2727.314617] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2727.323896] LustreError: 100864:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2727.329805] LustreError: 100864:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2727.336901] LustreError: 100864:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2727.387695] Lustre: server umount lustre-OST0000 complete [ 2727.390744] LustreError: 100864:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2736.424313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2736.663281] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2736.667629] Lustre: Found index 0 for lustre-OST0000, updating log [ 2736.674238] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2736.731165] Lustre: lustre-OST0000: new disk, initializing [ 2736.737339] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2736.748797] Lustre: 101398:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2737.127763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2743.593571] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2748.135326] Lustre: Failing over lustre-OST0000 [ 2748.370278] Lustre: server umount lustre-OST0000 complete [ 2753.517831] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2753.548954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2753.559253] Lustre: Skipped 2 previous similar messages [ 2754.600401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2754.612810] Lustre: Skipped 1 previous similar message [ 2759.651365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2759.656501] Lustre: Skipped 1 previous similar message [ 2764.772201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2764.784339] Lustre: Skipped 1 previous similar message [ 2766.304355] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2766.803884] Lustre: server umount lustre-MDT0000 complete [ 2769.889254] LustreError: 98314:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2769.925824] LustreError: 98314:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2771.014619] LustreError: 99054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787699906 with bad export cookie 7281191961309479326 [ 2771.030882] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2771.034585] LustreError: 99054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2771.484314] Lustre: server umount lustre-MDT0001 complete [ 2781.687376] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 19:18:34 (1787699914) [ 2803.085835] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 2814.244428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2814.759609] LustreError: 103746:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2814.897739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2819.914456] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2829.077137] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2834.416766] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2837.834646] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2845.698704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2846.642632] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2853.908978] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2854.833625] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2854.845422] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2854.867841] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2860.015658] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 2860.041240] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2860.071542] Lustre: lustre-OST0000: Recovery over after 0:06, of 1 clients 1 recovered and 0 were evicted. [ 2860.079331] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2860.090236] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2862.830150] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2862.934701] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2863.040889] Lustre: lustre-OST0001: new disk, initializing [ 2863.047597] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2863.055459] Lustre: 106064:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2863.353151] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2863.361038] Lustre: Skipped 2 previous similar messages [ 2869.920182] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2870.338505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2870.354108] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2870.453620] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2877.684401] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2887.143836] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2893.110520] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.44@tcp (stopping) [ 2896.353990] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2896.368703] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2896.407296] Lustre: Skipped 3 previous similar messages [ 2898.213597] LustreError: 103742:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2898.228212] LustreError: 103742:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2898.341729] Lustre: server umount lustre-MDT0001 complete [ 2909.074434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2909.215452] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2909.224245] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2909.290577] Lustre: 103736:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2911.028038] LustreError: 107618:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2911.049519] LustreError: 107618:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2911.074359] LustreError: 103738:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2911.212676] LustreError: 107634:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.144@tcp: cfg command failed: rc = -17 [ 2911.222878] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2911.222878] [ 2915.235872] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2915.327212] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2926.007079] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 19:20:59 (1787700059) [ 2943.990473] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 2962.305429] Lustre: 109682:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2988.655817] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2993.006667] Lustre: 110818:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3078.627718] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3078.639127] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3078.655322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3078.661451] Lustre: Skipped 3 previous similar messages [ 3081.599699] Lustre: server umount lustre-MDT0000 complete [ 3084.257476] LustreError: 103742:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3084.269053] LustreError: 103742:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 3085.039843] LustreError: 106077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787700220 with bad export cookie 7281191961309480327 [ 3085.040823] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3085.054192] LustreError: 106077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3085.357546] Lustre: server umount lustre-MDT0001 complete [ 3099.704140] Lustre: server umount lustre-OST0000 complete [ 3113.195442] Lustre: server umount lustre-OST0001 complete [ 3119.271794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3129.097987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3137.771134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3148.864203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3168.511688] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 3180.086528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3180.128612] Lustre: lustre-MDT0000: reset Object Index mappings [ 3180.809660] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3180.815228] Lustre: Skipped 1 previous similar message [ 3185.546628] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3193.604341] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3193.617937] Lustre: lustre-MDT0001: reset Object Index mappings [ 3198.490199] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3201.636164] Lustre: 115917:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3209.058106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3215.439607] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3218.997982] LustreError: 116827:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3219.018615] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 3219.027182] LustreError: 116827:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3223.442985] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3226.106879] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:382 to 0x280000401:418) [ 3226.116882] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3226.224149] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:199 to 0x2c0000400:225) [ 3230.841963] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3240.372610] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3244.390297] Lustre: 117795:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3293.257320] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 19:27:06 (1787700426) [ 3321.829405] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3321.845463] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3321.856888] Lustre: Skipped 3 previous similar messages [ 3321.862992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3321.868986] Lustre: Skipped 4 previous similar messages [ 3325.527903] Lustre: server umount lustre-MDT0000 complete [ 3329.383195] LustreError: 115519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787700465 with bad export cookie 7281191961309527430 [ 3329.384858] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3329.394420] LustreError: 115519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3333.602221] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3333.636092] Lustre: Skipped 4 previous similar messages [ 3336.014606] Lustre: server umount lustre-MDT0001 complete [ 3346.557498] Lustre: server umount lustre-OST0000 complete [ 3357.004473] Lustre: server umount lustre-OST0001 complete [ 3364.096579] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 3375.372880] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 3411.443862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3411.642632] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3411.678403] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3411.789587] Lustre: lustre-MDT0000: new disk, initializing [ 3411.862561] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3411.877593] Lustre: Skipped 3 previous similar messages [ 3411.908849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3416.192671] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3422.594968] Lustre: server umount lustre-MDT0000 complete [ 3432.071020] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3432.324542] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3432.334279] Lustre: 122060:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3432.350067] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3436.842306] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3441.569378] Lustre: server umount lustre-MDT0000 complete [ 3449.854173] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 19:29:43 (1787700583) [ 3451.300637] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3453.226550] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 19:29:46 (1787700586) [ 3470.709335] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 3481.360350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3487.883852] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3498.782071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3498.988769] Lustre: 124113:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3499.102801] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3499.111881] Lustre: Skipped 1 previous similar message [ 3499.254993] Lustre: lustre-MDT0001: new disk, initializing [ 3499.410823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3499.433872] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3506.115430] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3512.402336] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3519.722895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3520.118424] Lustre: lustre-OST0000: new disk, initializing [ 3520.122781] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3520.130772] Lustre: 125691:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3521.998083] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3522.013211] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3522.190846] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3526.864489] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3538.555933] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3538.715892] Lustre: lustre-OST0001: new disk, initializing [ 3538.719526] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3538.725670] Lustre: 126561:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3540.519102] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3540.539289] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3540.703623] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3547.180165] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3558.364715] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3562.312867] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3981.271103] Lustre: 132985:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3981.279111] Lustre: 132985:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3981.288766] Lustre: 132985:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3981.299789] Lustre: 132985:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3981.310925] Lustre: 132985:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3981.317867] Lustre: 132985:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4299.768971] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 19:43:51 (1787701431) [ 4319.202303] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4319.214305] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4319.237425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4319.254072] Lustre: Skipped 8 previous similar messages [ 4322.265557] Lustre: server umount lustre-MDT0000 complete [ 4325.352211] LustreError: 127382:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4325.392611] LustreError: 127382:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 4326.995948] LustreError: 133008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787701462 with bad export cookie 7281191961309595127 [ 4327.001951] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4327.009895] LustreError: 133008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4330.468773] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4330.483799] Lustre: Skipped 3 previous similar messages [ 4330.492528] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4330.495838] Lustre: Skipped 4 previous similar messages [ 4333.832074] Lustre: server umount lustre-MDT0001 complete [ 4344.554984] Lustre: server umount lustre-OST0000 complete [ 4354.921834] Lustre: server umount lustre-OST0001 complete [ 4364.576877] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 4377.093492] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 4415.495660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4415.744589] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4415.797513] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4415.916578] Lustre: lustre-MDT0000: new disk, initializing [ 4416.062232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4416.073361] Lustre: Skipped 5 previous similar messages [ 4416.101812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4421.603565] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 4435.357946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4435.660662] Lustre: 136043:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4435.751835] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4435.775104] Lustre: Skipped 1 previous similar message [ 4435.889781] Lustre: lustre-MDT0001: new disk, initializing [ 4436.168850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4436.183426] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4441.743499] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 4452.516988] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4458.248412] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4465.148226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4465.506208] Lustre: lustre-OST0000: new disk, initializing [ 4465.517328] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4465.530586] Lustre: 138012:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4465.652600] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4465.659501] Lustre: Skipped 1 previous similar message [ 4467.371965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4467.388651] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4467.554151] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4472.817886] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 4483.060902] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4493.566876] Lustre: lustre-MDD0000: changelog on [ 4496.686624] Lustre: lustre-MDD0001: changelog on [ 6301.113974] Lustre: lustre-MDD0001: changelog off [ 6302.760640] Lustre: 136050:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6303.684545] Lustre: lustre-MDD0000: changelog off [ 6310.890630] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6310.895047] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6310.904157] LustreError: Skipped 1 previous similar message [ 6310.914423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6310.914431] Lustre: Skipped 2 previous similar messages [ 6310.946742] Lustre: Skipped 2 previous similar messages [ 6316.005787] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6316.021119] Lustre: Skipped 2 previous similar messages [ 6321.122428] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6321.132472] Lustre: Skipped 1 previous similar message [ 6324.192157] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6324.520467] Lustre: server umount lustre-OST0000 complete [ 6330.337653] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6330.352681] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6330.369596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6331.379429] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6335.482690] Lustre: server umount lustre-MDT0000 complete [ 6336.496733] LustreError: 136853:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6336.526330] LustreError: 136853:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6339.878271] LustreError: 136037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787703475 with bad export cookie 7281191961310412237 [ 6339.899622] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6339.899849] LustreError: 136037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6340.447439] Lustre: server umount lustre-MDT0001 complete [ 6352.348226] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 6355.218592] Key type lgssc unregistered [ 6355.547970] LNet: 147148:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6355.554465] LNetError: 147148:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6355.570861] LNet: Removed LNI 192.168.203.144@tcp [ 6356.488258] Key type .llcrypt unregistered [ 6356.490310] Key type ._llcrypt unregistered [ 6377.524976] Key type ._llcrypt registered [ 6377.530112] Key type .llcrypt registered [ 6377.662883] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 6394.012788] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 6395.264609] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6395.416674] alg: No test for adler32 (adler32-zlib) [ 6396.575711] Lustre: Lustre: Build Version: 2.17.57_80_g4b309f5 [ 6396.894323] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 6398.640501] Key type lgssc registered [ 6400.003772] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6437.015744] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6437.044773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6438.411339] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6438.454549] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6438.582166] Lustre: lustre-MDT0000: new disk, initializing [ 6438.690119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6438.710434] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6443.347449] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6454.347666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6454.515777] Lustre: 150352:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6454.632679] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6454.640625] Lustre: Skipped 1 previous similar message [ 6454.775693] Lustre: lustre-MDT0001: new disk, initializing [ 6454.939667] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6455.008286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6455.023560] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6459.127744] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6467.814624] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6473.681854] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6481.114747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6481.570326] Lustre: lustre-OST0000: new disk, initializing [ 6481.579573] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6481.601770] Lustre: 152322:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6481.767643] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6490.042599] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6490.746066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6490.768893] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6490.954722] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6502.365945] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6508.965125] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_module obdecho/obdecho [ 6510.777552] LustreError: 153328:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6510.792466] LustreError: 153328:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6512.497680] LustreError: 153424:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6526.439220] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6526.458319] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6526.493158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6526.959909] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6526.993249] Lustre: Skipped 1 previous similar message [ 6527.000869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6527.014626] Lustre: Skipped 1 previous similar message [ 6531.536038] Lustre: server umount lustre-MDT0000 complete [ 6535.073975] LustreError: 150344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787703670 with bad export cookie 14690735891700153688 [ 6535.080968] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6535.092510] LustreError: 150344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6535.400039] Lustre: server umount lustre-MDT0001 complete [ 6548.409603] Lustre: server umount lustre-OST0000 complete [ 6566.286186] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 6575.950846] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 6616.828624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6617.051450] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6617.078885] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6617.230241] Lustre: lustre-MDT0000: new disk, initializing [ 6617.329770] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6617.355832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6621.745436] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6632.952460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6633.061923] Lustre: 157139:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6633.127095] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6633.131453] Lustre: Skipped 1 previous similar message [ 6633.213667] Lustre: lustre-MDT0001: new disk, initializing [ 6633.335441] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6633.377776] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6633.388920] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6638.048476] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6647.097934] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6652.560259] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6659.098391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6659.387341] Lustre: lustre-OST0000: new disk, initializing [ 6659.390319] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6659.397445] Lustre: 159109:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6659.501208] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6665.401612] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6669.390628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6669.409292] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6669.559523] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6675.689642] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6680.598497] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6680.951735] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6685.404696] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6685.580802] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6690.276277] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6690.281668] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6690.286830] LustreError: Skipped 1 previous similar message [ 6690.289684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6690.312177] Lustre: Skipped 1 previous similar message [ 6690.340945] Lustre: Skipped 1 previous similar message [ 6693.991585] Lustre: server umount lustre-OST0000 complete [ 6700.002713] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6700.015735] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6700.039941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6704.907847] Lustre: server umount lustre-MDT0000 complete [ 6705.635836] LustreError: 157153:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6705.655700] LustreError: 157153:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6708.916301] LustreError: 157132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787703844 with bad export cookie 14690735891700154647 [ 6708.920332] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6708.944974] LustreError: 157132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6709.416142] Lustre: server umount lustre-MDT0001 complete [ 6713.507864] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 20:24:06 (1787703846) [ 6715.008826] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6716.770443] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 20:24:10 (1787703850) [ 6724.708362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6725.163786] LustreError: 161522:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6725.226977] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6729.813720] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6730.735164] LustreError: 161523:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6735.848050] LustreError: 161522:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6738.918719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6743.968216] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6751.495499] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6757.145259] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6764.639378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6764.974835] LustreError: 163342:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6765.000953] LustreError: 163342:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6765.104663] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6765.112822] Lustre: Skipped 1 previous similar message [ 6771.808828] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6780.184694] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6786.536655] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6786.543772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6786.559821] Lustre: Skipped 2 previous similar messages [ 6786.582560] Lustre: Skipped 3 previous similar messages [ 6786.715760] Lustre: server umount lustre-MDT0000 complete [ 6790.393181] LustreError: 161503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787703926 with bad export cookie 14690735891700155529 [ 6790.405334] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6790.411087] LustreError: 161503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6790.966283] LustreError: 161519:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6795.681254] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6795.688873] Lustre: Skipped 2 previous similar messages [ 6797.050874] Lustre: server umount lustre-MDT0001 complete [ 6804.155956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6816.888402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6817.148229] LustreError: 165366:0:(llog.c:1655:llog_backup()) MGC192.168.203.144@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6817.155298] Lustre: 165366:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.144@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6820.322284] LustreError: 165371:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6820.356160] LustreError: 165371:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6856.481160] LustreError: 165371:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6856.518647] LustreError: 165371:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 30 previous similar messages [ 6877.664311] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6877.670246] Lustre: 165366:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.144@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6918.675238] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6922.012026] LustreError: 165372:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6922.057782] LustreError: 165372:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 43 previous similar messages [ 6923.756447] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6923.838214] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6934.335396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6936.677613] LustreError: 166108:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 6936.695304] LustreError: 166108:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@00000000cdf2020b x1874543018893312/t0(0) o250->MGC192.168.203.144@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1787704072 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6936.739386] LustreError: 166108:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 6937.060663] LustreError: 148573:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000056981953 x1874543018898944/t0(0) o250->MGC192.168.203.144@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6937.375895] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.44@tcp (not set up) [ 6937.402676] Lustre: Skipped 1 previous similar message [ 6942.713869] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6943.468126] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6951.974576] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6953.433826] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6959.286459] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6961.033126] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6983.649561] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6983.658870] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6983.667819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6983.690346] Lustre: Skipped 4 previous similar messages [ 6983.703528] Lustre: Skipped 2 previous similar messages [ 6984.483320] Lustre: server umount lustre-MDT0000 complete [ 6988.379160] LustreError: 164304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787704124 with bad export cookie 14690735891700161983 [ 6988.387181] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6988.390461] LustreError: 164304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6988.906088] Lustre: server umount lustre-MDT0001 complete [ 6995.339285] Lustre: server umount lustre-OST0000 complete [ 7021.103109] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 7033.423516] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 7071.268547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7071.606688] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7071.663346] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7071.776090] Lustre: lustre-MDT0000: new disk, initializing [ 7071.931806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7071.946503] Lustre: Skipped 1 previous similar message [ 7071.970433] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7076.537082] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7087.590414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7087.757273] Lustre: 170469:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7088.168114] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7092.726158] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7102.619450] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7108.298883] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7114.945802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7115.182591] Lustre: lustre-OST0000: new disk, initializing [ 7115.190557] Lustre: Skipped 1 previous similar message [ 7115.193849] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7115.205499] Lustre: Skipped 2 previous similar messages [ 7115.215823] Lustre: 172439:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7121.459941] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7123.522911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7123.534053] Lustre: Skipped 1 previous similar message [ 7123.544198] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7123.606252] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7130.621957] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7135.021531] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7135.222713] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7139.175649] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7139.384394] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7144.429675] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7144.449157] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7144.450532] Lustre: Skipped 1 previous similar message [ 7144.452365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7147.701165] Lustre: server umount lustre-OST0000 complete [ 7157.551817] Lustre: server umount lustre-MDT0000 complete [ 7159.777671] LustreError: 170480:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7159.791201] LustreError: 170480:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [ 7160.972838] LustreError: 170460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787704296 with bad export cookie 14690735891700178818 [ 7160.984589] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7160.992889] LustreError: 170460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7161.296270] Lustre: server umount lustre-MDT0001 complete [ 7169.330276] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 20:31:42 (1787704302) [ 7176.224646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7181.457722] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7189.606635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7194.145093] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7201.474965] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7207.712830] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7214.974750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7221.832986] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7230.856038] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7245.612759] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 20:32:58 (1787704378) [ 7250.401707] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7250.408157] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7250.433340] Lustre: Skipped 4 previous similar messages [ 7250.450648] LustreError: Skipped 1 previous similar message [ 7255.776867] Lustre: server umount lustre-OST0000 complete [ 7265.829046] Lustre: server umount lustre-MDT0000 complete [ 7269.416135] LustreError: 174828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787704405 with bad export cookie 14690735891700179700 [ 7269.419761] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7269.428629] LustreError: 174828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7269.721528] Lustre: server umount lustre-MDT0001 complete [ 7280.383225] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 7283.196800] Key type lgssc unregistered [ 7283.556578] LNet: 178580:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7283.576434] LNetError: 178580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7283.598485] LNet: Removed LNI 192.168.203.144@tcp [ 7284.540575] Key type .llcrypt unregistered [ 7284.544111] Key type ._llcrypt unregistered [ 7303.160578] Key type ._llcrypt registered [ 7303.163601] Key type .llcrypt registered [ 7303.290837] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 7304.252340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7304.374309] alg: No test for adler32 (adler32-zlib) [ 7305.576904] Lustre: Lustre: Build Version: 2.17.57_80_g4b309f5 [ 7305.785555] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 7307.443112] Key type lgssc registered [ 7308.675553] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7318.199269] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7318.239043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7334.048521] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7339.168334] LustreError: 179823:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.144@tcp: failed processing log, type 4: rc = -110 [ 7344.288442] LustreError: 179823:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7344.299916] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7344.303942] LustreError: MGC192.168.203.144@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7344.322259] LustreError: 179823:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7344.336231] LustreError: 179823:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -22 [ 7344.353120] LustreError: 179823:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 7344.439967] Lustre: server umount lustre-OST0000 complete [ 7344.447920] LustreError: 179823:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7353.773853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7355.177189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7359.284186] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7365.607977] LustreError: 180324:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7365.627892] LustreError: 180324:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7367.146273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7367.496302] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7371.634863] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7379.965567] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7386.465034] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7393.179087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7393.432709] LustreError: 182138:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7393.451649] Lustre: 182138:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.203.144@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7393.652753] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7400.196687] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7407.871546] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7411.171608] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7411.179101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7411.198636] Lustre: Skipped 1 previous similar message [ 7412.259733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7412.278288] Lustre: Skipped 4 previous similar messages [ 7416.543406] Lustre: server umount lustre-MDT0000 complete [ 7417.321874] LustreError: 180319:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7417.340658] LustreError: 180319:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7419.956723] LustreError: 180304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787704555 with bad export cookie 13989410393511378784 [ 7419.965396] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7419.969749] LustreError: 180304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7420.472331] LustreError: 181051:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000007942570c x1874543971954816/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7420.600341] Lustre: server umount lustre-MDT0001 complete [ 7434.765429] Lustre: server umount lustre-OST0000 complete [ 7442.998534] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 7451.668306] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 7485.216946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7485.432771] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7485.481172] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7485.618464] Lustre: lustre-MDT0000: new disk, initializing [ 7485.710754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7485.729689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7489.889380] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7499.531446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7499.709415] Lustre: 185662:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7499.784586] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7499.792846] Lustre: Skipped 1 previous similar message [ 7499.899531] Lustre: lustre-MDT0001: new disk, initializing [ 7500.074734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7500.122894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7500.144931] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7504.317500] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7512.381662] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7516.961870] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7523.077030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7523.333701] Lustre: lustre-OST0000: new disk, initializing [ 7523.336918] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7523.344765] Lustre: 187629:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7523.433721] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7529.127782] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7530.050418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7530.066872] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7530.167736] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7537.340464] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7541.435317] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7541.612786] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7545.366522] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7545.558766] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7550.946454] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7550.951707] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7550.959573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7550.974609] Lustre: Skipped 2 previous similar messages [ 7550.986275] Lustre: Skipped 1 previous similar message [ 7553.274922] Lustre: server umount lustre-OST0000 complete [ 7561.188904] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7561.200314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7561.209524] Lustre: Skipped 1 previous similar message [ 7563.233442] Lustre: server umount lustre-MDT0000 complete [ 7566.309306] LustreError: 185671:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7566.333209] LustreError: 185671:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7566.393533] LustreError: 185655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787704702 with bad export cookie 13989410393511379498 [ 7566.399072] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7566.400244] LustreError: 185655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7566.636750] Lustre: server umount lustre-MDT0001 complete [ 7573.922908] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 20:38:27 (1787704707) [ 7589.639810] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 7592.252725] Key type lgssc unregistered [ 7592.580160] LNet: 190575:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7592.592029] LNetError: 190575:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7592.621241] LNet: Removed LNI 192.168.203.144@tcp [ 7593.527867] Key type .llcrypt unregistered [ 7593.533116] Key type ._llcrypt unregistered [ 7610.951722] Key type ._llcrypt registered [ 7610.955401] Key type .llcrypt registered [ 7611.049354] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 7611.925119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7611.968127] alg: No test for adler32 (adler32-zlib) [ 7613.069169] Lustre: Lustre: Build Version: 2.17.57_80_g4b309f5 [ 7613.323326] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 7615.064252] Key type lgssc registered [ 7616.288075] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7623.411393] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7623.436562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7624.941544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7629.543632] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7634.405493] LustreError: 191793:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7634.419375] LustreError: 191793:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7638.302828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7638.668495] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7642.683829] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7650.108351] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7654.408343] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7660.766147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7660.976656] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7660.979568] LustreError: 193657:0:(llog.c:1696:llog_backup()) MGC192.168.203.144@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7660.995319] Lustre: 193657:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.203.144@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7661.103840] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7666.317752] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7672.324858] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7674.340411] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7674.341747] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7674.345426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7674.371805] Lustre: Skipped 1 previous similar message [ 7679.461239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7679.466386] Lustre: Skipped 2 previous similar messages [ 7680.446501] Lustre: server umount lustre-MDT0000 complete [ 7683.725672] LustreError: 191775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787704819 with bad export cookie 757662651368908979 [ 7683.729829] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7683.737666] LustreError: 191775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7684.051704] Lustre: server umount lustre-MDT0001 complete [ 7697.853080] Lustre: server umount lustre-OST0000 complete [ 7705.474626] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 7714.252321] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 7745.247261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7745.455750] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7745.491737] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7745.581662] Lustre: lustre-MDT0000: new disk, initializing [ 7745.652482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7745.667760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7749.267724] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7758.555385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7758.644342] Lustre: 197180:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7758.681198] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7758.683892] Lustre: Skipped 1 previous similar message [ 7758.686776] LustreError: 197963:0:(llog.c:1696:llog_backup()) MGC192.168.203.144@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7758.692820] LustreError: 197963:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7758.697030] Lustre: 197963:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.144@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7758.711849] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7758.715932] Lustre: Skipped 1 previous similar message [ 7758.780731] Lustre: lustre-MDT0001: new disk, initializing [ 7758.854971] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7758.885051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7758.898410] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7762.844053] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7770.917240] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7776.554539] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7782.577570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7782.846900] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7782.851263] Lustre: Skipped 1 previous similar message [ 7782.858171] LustreError: 199123:0:(llog.c:1696:llog_backup()) MGC192.168.203.144@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7782.871231] LustreError: 199123:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7782.880080] Lustre: 199123:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.144@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7782.888453] Lustre: 199123:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7782.911841] Lustre: lustre-OST0000: new disk, initializing [ 7782.915853] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7782.923537] Lustre: 199146:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7783.040545] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7788.746662] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7790.166726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7790.181448] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7790.318226] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7797.836465] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7801.955397] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7802.139499] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7805.886908] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7806.112569] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7811.044546] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7811.049300] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7811.050400] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7811.066560] Lustre: Skipped 2 previous similar messages [ 7811.098775] Lustre: Skipped 2 previous similar messages [ 7816.163538] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7821.284601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7821.289063] Lustre: Skipped 2 previous similar messages [ 7822.304263] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7822.539739] Lustre: server umount lustre-OST0000 complete [ 7830.497299] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7830.506360] LustreError: Skipped 1 previous similar message [ 7830.510718] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7830.539577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7841.249045] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7841.435829] Lustre: server umount lustre-MDT0000 complete [ 7841.763221] LustreError: 197186:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7841.791727] LustreError: 197186:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7845.180919] LustreError: 197171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787704981 with bad export cookie 757662651368909700 [ 7845.200695] LustreError: 197171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7845.209187] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7845.644321] Lustre: server umount lustre-MDT0001 complete [ 7854.510525] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 20:43:07 (1787704987) [ 7869.200170] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 7878.606774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7878.869613] LustreError: 202273:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7878.922839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7883.236875] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7884.275481] LustreError: 202274:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7888.356934] LustreError: 202273:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7890.970353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7896.243551] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7899.276787] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7906.307485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7912.511903] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7920.695773] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7920.798830] Lustre: lustre-OST0001: new disk, initializing [ 7920.802039] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7920.809418] Lustre: 204580:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7920.899350] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7920.904411] Lustre: Skipped 2 previous similar messages [ 7923.448678] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7923.459589] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7923.614897] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7927.520308] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7939.706255] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7947.453800] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7949.808783] Lustre: DEBUG MARKER: ADD OST3 [ 7954.502264] loop: module loaded [ 7954.571147] loop0: detected capacity change from 0 to 204800000 [ 7954.816698] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7954.825124] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7954.852132] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7960.365517] loop0: detected capacity change from 0 to 204800000 [ 7961.641156] Lustre: DEBUG MARKER: STOP OST3 [ 7965.412955] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7965.626710] Lustre: lustre-OST0003: new disk, initializing [ 7965.630779] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7965.635252] Lustre: 206541:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7967.728322] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7967.878572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7967.889982] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7967.928878] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7969.706955] Lustre: Failing over lustre-OST0003 [ 7969.763670] LustreError: 206788:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7970.064941] Lustre: server umount lustre-OST0003 complete [ 7971.724599] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7972.163225] LustreError: 203760:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.203.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7972.844307] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7972.854912] Lustre: Skipped 2 previous similar messages [ 7980.431383] Lustre: DEBUG MARKER: START OST3 again [ 7986.709745] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7986.948808] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7986.955944] Lustre: Skipped 1 previous similar message [ 7986.982341] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7987.485441] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7988.291438] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7988.292266] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7992.907915] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8015.351593] Lustre: Failing over lustre-OST0003 [ 8015.466373] Lustre: server umount lustre-OST0003 complete [ 8017.906413] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8017.928562] LustreError: 207310:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8017.942969] LustreError: 207310:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 8033.254901] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8033.257248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8033.275548] Lustre: Skipped 4 previous similar messages [ 8033.281899] Lustre: Skipped 6 previous similar messages [ 8038.676794] Lustre: server umount lustre-MDT0000 complete [ 8042.432341] LustreError: 202254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787705178 with bad export cookie 757662651368910582 [ 8042.450961] LustreError: 202254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8042.456841] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8048.927820] Lustre: server umount lustre-MDT0001 complete [ 8052.771836] Lustre: server umount lustre-OST0000 complete [ 8056.910196] Lustre: server umount lustre-OST0001 complete [ 8063.908355] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 8075.599166] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 8111.685834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8111.922500] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8111.966426] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8112.094181] Lustre: lustre-MDT0000: new disk, initializing [ 8112.182841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8116.994697] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8127.790771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8127.893971] Lustre: 212036:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8128.103701] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8128.108871] Lustre: Skipped 1 previous similar message [ 8128.146305] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8132.747974] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8142.618469] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8148.341787] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8154.732332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8154.953533] Lustre: lustre-OST0000: new disk, initializing [ 8154.956479] Lustre: Skipped 1 previous similar message [ 8154.961237] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8154.965881] Lustre: Skipped 2 previous similar messages [ 8154.972368] Lustre: 214005:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8156.387116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8156.412234] Lustre: Skipped 1 previous similar message [ 8156.414738] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8156.553698] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8160.786927] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8170.852165] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8176.655110] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8176.966986] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8183.202991] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8183.529461] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8187.365267] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8187.379840] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8187.391795] Lustre: Skipped 2 previous similar messages [ 8187.413389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8187.422247] Lustre: Skipped 9 previous similar messages [ 8191.674149] Lustre: server umount lustre-OST0000 complete [ 8202.726966] LustreError: 212048:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8202.752894] LustreError: 212048:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 8202.894188] Lustre: server umount lustre-MDT0000 complete [ 8207.561460] LustreError: 212029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787705343 with bad export cookie 757662651368912871 [ 8207.567869] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8207.583947] LustreError: 212029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8208.102350] Lustre: server umount lustre-MDT0001 complete [ 8212.214467] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 20:49:05 (1787705345) [ 8227.447271] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 8237.539413] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 8270.921798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8271.217893] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8271.252726] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8271.346924] Lustre: lustre-MDT0000: new disk, initializing [ 8271.442619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8275.658378] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8285.539795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8285.618364] Lustre: 218615:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8285.806229] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8290.341358] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8298.875672] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8304.144728] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8310.962520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8311.315763] Lustre: 220583:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8312.555354] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8318.399615] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8329.061223] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8334.247817] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8334.490152] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8339.226493] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8339.520096] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8343.526452] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8343.540102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8343.545317] Lustre: Skipped 4 previous similar messages [ 8343.548224] Lustre: Skipped 2 previous similar messages [ 8343.557450] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8347.781280] Lustre: server umount lustre-OST0000 complete [ 8352.225492] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8357.898046] Lustre: server umount lustre-MDT0000 complete [ 8358.883542] LustreError: 219510:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8358.900514] LustreError: 219510:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8361.655118] LustreError: 218605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787705497 with bad export cookie 757662651368913753 [ 8361.668138] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8361.673756] LustreError: 218605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8361.923044] Lustre: server umount lustre-MDT0001 complete [ 8370.614731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8376.334988] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8384.882571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8385.166810] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8385.173189] Lustre: Skipped 5 previous similar messages [ 8389.709562] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8397.507396] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8405.439274] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8413.505224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8421.707227] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8429.091349] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8440.813154] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8440.819298] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8440.829729] Lustre: Skipped 3 previous similar messages [ 8445.424594] Lustre: server umount lustre-OST0000 complete [ 8455.451339] Lustre: server umount lustre-MDT0000 complete [ 8459.262742] LustreError: 222780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787705595 with bad export cookie 757662651368914635 [ 8459.273988] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8459.280341] LustreError: 222780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8459.723447] Lustre: server umount lustre-MDT0001 complete [ 8568.867188] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 8571.736909] Key type lgssc unregistered [ 8572.082891] LNet: 226468:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8572.097619] LNetError: 226468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8572.119379] LNet: Removed LNI 192.168.203.144@tcp [ 8573.113276] Key type .llcrypt unregistered [ 8573.117921] Key type ._llcrypt unregistered [ 8599.918297] Key type ._llcrypt registered [ 8599.920699] Key type .llcrypt registered [ 8600.048536] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 8619.473481] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 8620.815503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8620.851080] alg: No test for adler32 (adler32-zlib) [ 8621.953984] Lustre: Lustre: Build Version: 2.17.57_80_g4b309f5 [ 8622.180486] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 8623.896265] Key type lgssc registered [ 8625.026602] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8662.564342] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8662.598275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8664.020983] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8664.102623] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8664.284677] Lustre: lustre-MDT0000: new disk, initializing [ 8664.421510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8664.447320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8671.471657] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8682.741539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8682.840146] Lustre: 229820:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8682.903851] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8682.907463] Lustre: Skipped 1 previous similar message [ 8682.971197] Lustre: lustre-MDT0001: new disk, initializing [ 8683.054096] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8683.101738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8683.114264] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8688.526125] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8698.531331] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8705.208796] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8713.531923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8713.817739] Lustre: lustre-OST0000: new disk, initializing [ 8713.823171] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8713.829905] Lustre: 231794:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8713.985637] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8720.972595] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8722.039481] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8722.051881] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8722.255927] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8731.405984] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8736.747605] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8736.918631] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8742.147911] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8742.385369] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8748.005649] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8748.006416] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8748.016729] LustreError: Skipped 1 previous similar message [ 8748.025444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8748.035719] Lustre: Skipped 1 previous similar message [ 8748.065152] Lustre: Skipped 1 previous similar message [ 8750.701278] Lustre: server umount lustre-OST0000 complete [ 8758.250483] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8758.261492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8758.275920] Lustre: Skipped 1 previous similar message [ 8758.289593] Lustre: Skipped 1 previous similar message [ 8761.477961] Lustre: server umount lustre-MDT0000 complete [ 8765.625140] LustreError: 231793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787705901 with bad export cookie 1598702417254110728 [ 8765.631985] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8765.639276] LustreError: 231793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8765.922500] Lustre: server umount lustre-MDT0001 complete [ 8785.132643] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 8795.207900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8795.510211] LustreError: 234727:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8795.535210] LustreError: 234727:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8795.590566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8800.160464] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8800.739525] LustreError: 234728:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8805.866317] LustreError: 234727:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8809.181361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8809.627390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8814.958938] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8818.537296] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8825.599849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8826.081349] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8833.905760] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8844.090248] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8844.346514] Lustre: lustre-OST0001: new disk, initializing [ 8844.357536] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8844.365847] Lustre: 237036:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8844.486787] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8851.611925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8851.617682] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8851.764863] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8852.378172] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8864.637973] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8869.383409] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8881.987239] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 21:00:15 (1787706015) [ 8886.241413] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8886.251463] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8886.266698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8890.250565] Lustre: server umount lustre-MDT0000 complete [ 8892.901817] LustreError: 234722:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8892.920805] LustreError: 234722:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8893.787656] LustreError: 237804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787706029 with bad export cookie 1598702417254111603 [ 8893.796872] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8893.806196] LustreError: 237804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8894.160986] Lustre: server umount lustre-MDT0001 complete [ 8907.458440] Lustre: server umount lustre-OST0000 complete [ 8922.156374] Lustre: server umount lustre-OST0001 complete [ 8930.053489] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 8941.906472] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 8982.301572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8982.533500] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8982.556456] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8982.620626] Lustre: lustre-MDT0000: new disk, initializing [ 8982.707826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8982.724950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8987.410364] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8998.729036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8998.864744] Lustre: 240995:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8998.933550] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8998.936859] Lustre: Skipped 1 previous similar message [ 8999.029777] Lustre: lustre-MDT0001: new disk, initializing [ 8999.181335] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8999.220133] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9004.323285] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9015.221696] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9021.626801] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9029.844455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9030.415328] Lustre: lustre-OST0000: new disk, initializing [ 9030.436206] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9030.456907] Lustre: 242967:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9032.132584] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9032.149290] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9037.319103] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9037.921973] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9049.183180] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9055.716817] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9056.056228] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9062.405760] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9062.675966] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9068.012391] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9068.014880] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9068.044076] LustreError: Skipped 1 previous similar message [ 9068.059370] Lustre: Skipped 3 previous similar messages [ 9068.062376] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9068.088654] Lustre: Skipped 5 previous similar messages [ 9071.375586] Lustre: server umount lustre-OST0000 complete [ 9076.194068] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9076.203823] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9076.221945] Lustre: Skipped 1 previous similar message [ 9076.253488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9082.043376] Lustre: server umount lustre-MDT0000 complete [ 9083.363624] LustreError: 241007:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9083.389578] LustreError: 241007:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9086.347270] LustreError: 240986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787706222 with bad export cookie 1598702417254113752 [ 9086.348873] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9086.363297] LustreError: 240986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9086.850654] Lustre: server umount lustre-MDT0001 complete [ 9095.860244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9096.422845] LustreError: 245183:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9096.515599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9096.519086] Lustre: Skipped 2 previous similar messages [ 9102.160522] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9111.536562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9117.018497] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9125.143710] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9131.003945] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9138.348574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9145.765821] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9153.453292] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9305.057631] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9305.075718] Lustre: Skipped 1 previous similar message [ 9305.078569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9305.095524] Lustre: Skipped 3 previous similar messages [ 9305.100364] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9307.449977] Lustre: server umount lustre-OST0000 complete [ 9325.548206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9325.560487] Lustre: Skipped 5 previous similar messages [ 9326.048143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9326.517952] Lustre: server umount lustre-MDT0000 complete [ 9330.660895] LustreError: 245178:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9330.679496] LustreError: 245178:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9332.570280] LustreError: 245163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787706468 with bad export cookie 1598702417254114634 [ 9332.584736] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9333.295593] Lustre: server umount lustre-MDT0001 complete [ 9420.913787] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 9423.937566] Key type lgssc unregistered [ 9424.235933] LNet: 248795:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9424.246183] LNetError: 248795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9424.262843] LNet: Removed LNI 192.168.203.144@tcp [ 9425.162283] Key type .llcrypt unregistered [ 9425.169096] Key type ._llcrypt unregistered [ 9457.032503] Key type ._llcrypt registered [ 9457.034334] Key type .llcrypt registered [ 9457.208654] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 9473.988719] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 9475.070411] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9475.090029] alg: No test for adler32 (adler32-zlib) [ 9476.263885] Lustre: Lustre: Build Version: 2.17.57_80_g4b309f5 [ 9476.596580] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 9478.313567] Key type lgssc registered [ 9479.631564] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9525.701242] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 9536.791758] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9536.833376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9538.127789] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9538.159907] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9538.249577] Lustre: lustre-MDT0000: new disk, initializing [ 9538.351653] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9538.377120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9543.059237] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9554.115110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9554.240962] Lustre: 252870:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9554.324887] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9554.328862] Lustre: Skipped 1 previous similar message [ 9554.437450] Lustre: lustre-MDT0001: new disk, initializing [ 9554.558784] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9554.594871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9554.612095] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9559.823433] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9565.106765] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9572.054291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9572.393350] Lustre: lustre-OST0000: new disk, initializing [ 9572.402327] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9572.416204] Lustre: 254501:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9572.500202] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9579.944958] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9582.189166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9582.213750] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9582.321349] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9591.289635] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9591.485315] Lustre: lustre-OST0001: new disk, initializing [ 9591.490779] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9591.497858] Lustre: 255375:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9591.593786] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9598.183256] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9600.647602] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9600.659383] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9600.778937] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9608.576208] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9618.294912] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9630.834334] Lustre: server umount lustre-MDT0000 complete [ 9631.734858] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9631.740744] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9631.753201] Lustre: Skipped 3 previous similar messages [ 9634.211291] LustreError: 254500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787706770 with bad export cookie 7633880902582862927 [ 9634.223581] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9634.235782] LustreError: 254500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9634.912142] Lustre: server umount lustre-MDT0001 complete [ 9649.247630] Lustre: server umount lustre-OST0000 complete [ 9652.196114] Lustre: 250377:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787706772/real 1787706772] req@ffff96f101951880 x1874546248460928/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787706788 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9652.247889] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9654.332703] Lustre: server umount lustre-OST0001 complete [ 9672.937955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9684.586374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9693.998183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9706.188186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9728.211663] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 9739.569362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9739.593829] Lustre: lustre-MDT0000: reset Object Index mappings [ 9740.202419] LustreError: 260382:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9740.239789] LustreError: 260382:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9740.345570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9745.115952] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9745.380099] LustreError: 260383:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9750.506197] LustreError: 260382:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9755.222096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9755.256840] Lustre: lustre-MDT0001: reset Object Index mappings [ 9755.618795] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9755.916676] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9759.828976] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9763.450823] Lustre: 261525:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9770.079040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9776.644405] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9780.671700] LustreError: 261877:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9785.786662] LustreError: 261877:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9785.811996] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9785.814557] LustreError: 261877:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9786.420135] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9786.628712] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9786.631526] Lustre: Skipped 1 previous similar message [ 9792.013949] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9792.381082] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9801.052644] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9804.601130] Lustre: 263397:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9810.261744] Lustre: server umount lustre-MDT0000 complete [ 9811.936921] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9811.950268] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9811.973255] LustreError: 261445:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9814.502072] LustreError: 263399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787706950 with bad export cookie 7633880902582869717 [ 9814.508668] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9814.515208] LustreError: 263399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9814.983712] Lustre: server umount lustre-MDT0001 complete [ 9829.733388] Lustre: server umount lustre-OST0000 complete [ 9843.330982] Lustre: server umount lustre-OST0001 complete [ 9873.800695] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 21:16:46 (1787707006) [ 9894.364987] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 9906.260734] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 9944.657548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9944.889156] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9944.915810] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9944.968366] Lustre: lustre-MDT0000: new disk, initializing [ 9945.016462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9945.027794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9949.057076] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9960.649709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9960.791377] Lustre: 267657:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9960.848766] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9960.855507] Lustre: Skipped 1 previous similar message [ 9960.935741] Lustre: lustre-MDT0001: new disk, initializing [ 9961.094602] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9961.114702] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9966.488648] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9976.852473] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9982.748776] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9990.399800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9990.643306] Lustre: lustre-OST0000: new disk, initializing [ 9990.647709] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9990.653361] Lustre: 269628:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9992.330927] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9992.353620] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9992.498741] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9996.777970] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10008.375963] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10013.885097] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10014.242621] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10019.463155] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10019.815661] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10023.399798] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10023.412815] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10023.441576] Lustre: Skipped 3 previous similar messages [10023.446038] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10023.456445] Lustre: Skipped 1 previous similar message [10028.358201] Lustre: server umount lustre-OST0000 complete [10033.633479] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10033.643779] Lustre: Skipped 1 previous similar message [10033.648093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10033.659192] Lustre: Skipped 1 previous similar message [10038.754224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10038.759744] Lustre: Skipped 1 previous similar message [10039.667528] Lustre: server umount lustre-MDT0000 complete [10043.880314] LustreError: 268932:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10043.916077] LustreError: 268932:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10044.717774] LustreError: 269627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787707180 with bad export cookie 7633880902582871810 [10044.718721] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10044.737898] LustreError: 269627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10045.226179] Lustre: server umount lustre-MDT0001 complete [10069.982390] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [10082.262325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10082.756774] LustreError: 272584:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10082.839300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10082.844905] Lustre: Skipped 2 previous similar messages [10088.908832] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10100.978372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10105.841928] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10109.561463] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10117.418967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10126.384445] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10135.529775] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10135.683725] Lustre: lustre-OST0001: new disk, initializing [10135.687815] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10135.694147] Lustre: 274894:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10136.971534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10136.995762] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10137.202298] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10142.301168] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10152.194354] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10156.059757] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10161.104270] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10161.265589] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10167.789099] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10167.797445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10167.814542] Lustre: Skipped 3 previous similar messages [10167.825038] Lustre: Skipped 3 previous similar messages [10175.446133] Lustre: server umount lustre-MDT0000 complete [10178.023513] LustreError: 272580:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10178.067178] LustreError: 272580:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [10179.451884] LustreError: 274094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787707315 with bad export cookie 7633880902582872692 [10179.463247] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10179.464711] LustreError: 274094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10179.944632] Lustre: server umount lustre-MDT0001 complete [10194.900178] Lustre: server umount lustre-OST0000 complete [10209.169849] Lustre: server umount lustre-OST0001 complete [10227.285500] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [10238.604168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10239.203647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10239.216235] Lustre: Skipped 3 previous similar messages [10244.227480] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10252.981147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10258.336446] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10261.968362] Lustre: 279092:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10268.814139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10277.280887] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10285.077053] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10288.186696] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10293.741945] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10295.177063] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10303.324434] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10308.060933] Lustre: 280970:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10314.142833] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10314.170359] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10324.822428] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 21:24:17 (1787707457) [10344.941200] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10344.952167] Lustre: Skipped 2 previous similar messages [10344.960240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10344.976717] Lustre: Skipped 7 previous similar messages [10355.168137] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10355.383913] Lustre: server umount lustre-MDT0000 complete [10360.296967] LustreError: 277946:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10360.349766] LustreError: 277946:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [10361.198158] LustreError: 277932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787707497 with bad export cookie 7633880902582879034 [10361.202774] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10361.212176] LustreError: 277932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10365.418734] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10365.421475] Lustre: Skipped 8 previous similar messages [10367.761640] Lustre: server umount lustre-MDT0001 complete [10378.950061] Lustre: server umount lustre-OST0000 complete [10383.293747] Lustre: server umount lustre-OST0001 complete [10390.993735] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [10402.594628] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [10451.243592] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [10462.812953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10462.985735] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10463.015450] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10463.075440] Lustre: lustre-MDT0000: new disk, initializing [10463.141534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10467.940746] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10479.114904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10479.273607] Lustre: 285209:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10479.571233] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10484.373143] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10498.041170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10498.333176] Lustre: 286843:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10498.433218] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10498.443396] Lustre: Skipped 5 previous similar messages [10500.558539] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10505.035159] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10516.084336] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10516.345570] Lustre: 287716:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10517.916147] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10517.929310] Lustre: Skipped 1 previous similar message [10518.020561] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10523.007110] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10532.791406] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10536.506230] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10536.516635] Lustre: Skipped 1 previous similar message [10551.377263] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 21:28:04 (1787707684) [10553.826629] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10553.831846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10553.860490] Lustre: Skipped 6 previous similar messages [10553.886145] Lustre: Skipped 5 previous similar messages [10555.883429] Lustre: server umount lustre-MDT0000 complete [10559.704245] LustreError: 285203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787707695 with bad export cookie 7633880902582886909 [10559.710285] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10559.728293] LustreError: 285203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10560.313743] Lustre: server umount lustre-MDT0001 complete [10573.487030] Lustre: server umount lustre-OST0000 complete [10586.394885] Lustre: server umount lustre-OST0001 complete [10594.766949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10600.260789] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10609.103751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10614.367928] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10621.623428] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10627.448652] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10634.186394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10640.748104] LustreError: 291793:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10640.775213] LustreError: 291793:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [10641.121379] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10648.707391] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10668.862026] loop1: detected capacity change from 0 to 204800000 [10669.235541] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10669.259328] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10669.283115] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10675.033514] loop1: detected capacity change from 0 to 204800000 [10681.591249] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10681.765117] LustreError: 289952:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs27822: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10681.786586] LustreError: 289952:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs27822 [10681.796707] LustreError: 289952:0:(mgs_handler.c:517:mgs_target_reg()) fs27822-MDT0001: Error checking target index: rc = -13 [10681.826119] LustreError: fs27822-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10681.841405] LustreError: 293684:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [10681.856805] LustreError: 293684:0:(tgt_mount.c:2132:server_put_super()) no obd fs27822-MDT0001 [10681.867000] LustreError: 293684:0:(tgt_mount.c:116:server_deregister_mount()) fs27822-MDT0001 not registered [10681.923801] Lustre: server umount fs27822-MDT0001 complete [10681.930058] LustreError: 293684:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10692.466742] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10692.704038] Lustre: Setting parameter fs27822-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs27822-MDT0001 [10692.787508] Lustre: srv-fs27822-MDT0001: No data found on store. Initialize space. [10692.792591] Lustre: Skipped 4 previous similar messages [10692.903467] Lustre: fs27822-MDT0001: new disk, initializing [10692.908119] Lustre: Skipped 3 previous similar messages [10693.079143] Lustre: srv-fs27822-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10693.099537] Lustre: 294150:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs27822-MDT0001-mdtlov: can't create new llog: rc = -115 [10693.125781] LustreError: 294150:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs27822-MDT0001-osd: get update log duration 1, retries 0, failed: rc = -115 [10698.763720] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10705.994029] Lustre: server umount fs27822-MDT0001 complete [10720.599465] loop1: detected capacity change from 0 to 204800000 [10720.746198] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10720.772028] LustreError: Skipped 1 previous similar message [10726.504178] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10726.978047] Lustre: 295719:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs27822-MDT0001-mdtlov: can't create new llog: rc = -115 [10726.986969] LustreError: 295719:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs27822-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10731.734358] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10736.099558] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10736.544231] Lustre: server umount fs27822-MDT0001 complete [10746.337845] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10746.340412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10746.358527] Lustre: Skipped 2 previous similar messages [10746.375251] Lustre: Skipped 2 previous similar messages [10752.571039] Lustre: server umount lustre-MDT0000 complete [10761.379834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10761.568982] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10766.538828] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10767.363278] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10774.106546] loop1: detected capacity change from 0 to 204800000 [10779.397774] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10779.712595] Lustre: 298164:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs27822-MDT0001-mdtlov: can't create new llog: rc = -115 [10779.725122] LustreError: 298164:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs27822-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10783.545102] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10787.989399] Lustre: server umount fs27822-MDT0001 complete [10814.267958] Lustre: server umount lustre-OST0000 complete [10818.529197] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10832.353854] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10832.648320] Lustre: server umount lustre-MDT0000 complete [10836.366572] LustreError: 289943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787707972 with bad export cookie 7633880902582889590 [10836.369775] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10836.377159] LustreError: 289943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10836.724638] Lustre: server umount lustre-MDT0001 complete [10846.914717] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [10849.429995] Key type lgssc unregistered [10849.703781] LNet: 300511:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10849.712542] LNetError: 300511:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10849.739464] LNet: Removed LNI 192.168.203.144@tcp [10850.548156] Key type .llcrypt unregistered [10850.552057] Key type ._llcrypt unregistered [10871.216022] Key type ._llcrypt registered [10871.217687] Key type .llcrypt registered [10871.398239] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [10873.306990] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10873.363077] alg: No test for adler32 (adler32-zlib) [10874.435768] Lustre: Lustre: Build Version: 2.17.57_80_g4b309f5 [10874.752082] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [10876.536160] Key type lgssc registered [10878.372325] Lustre: Echo OBD driver; http://www.lustre.org/ [10887.813408] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10887.841544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10889.302885] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10894.443440] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10899.441141] LustreError: 301784:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10899.463305] LustreError: 301784:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10903.586131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10903.997255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10908.592891] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10915.025783] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10920.594847] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10927.330451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10927.632381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10928.691033] LustreError: 303613:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10933.740284] LustreError: 303613:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10934.885059] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10939.894126] LustreError: 303614:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10939.913044] LustreError: 303614:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10943.801488] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10944.996883] LustreError: 303614:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10945.042252] LustreError: 303614:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10955.256027] LustreError: 303614:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10955.312523] LustreError: 303614:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10964.604059] loop1: detected capacity change from 0 to 204800000 [10964.816830] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10964.837325] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10964.860363] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10970.716751] loop1: detected capacity change from 0 to 204800000 [10972.447377] LustreError: 303613:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10972.492142] LustreError: 303613:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [10978.176371] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10978.321874] Lustre: fs26182-OST0000: new disk, initializing [10978.325696] Lustre: srv-fs26182-OST0000: No data found on store. Initialize space. [10978.332171] Lustre: 305520:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10978.430472] Lustre: fs26182-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10984.253168] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10992.360218] Lustre: server umount fs26182-OST0000 complete [11004.949974] loop1: detected capacity change from 0 to 204800000 [11006.437494] LustreError: 303613:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11006.464064] LustreError: 303613:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 19 previous similar messages [11010.066848] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11010.352313] Lustre: fs26182-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11011.560535] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11016.697720] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11017.083296] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11025.207908] loop2: detected capacity change from 0 to 204800000 [11025.425803] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11025.437854] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11025.446410] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11030.465705] loop2: detected capacity change from 0 to 204800000 [11035.717775] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11035.794176] LustreError: fs26182-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11035.805573] LustreError: fs26182-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11035.815528] LustreError: 308558:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [11035.821968] LustreError: 308558:0:(tgt_mount.c:2132:server_put_super()) no obd fs26182-OST0001 [11035.827267] LustreError: 308558:0:(tgt_mount.c:116:server_deregister_mount()) fs26182-OST0001 not registered [11035.856765] Lustre: server umount fs26182-OST0001 complete [11035.861394] LustreError: 308558:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11046.090385] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11046.332703] Lustre: fs26182-OST0001: new disk, initializing [11046.342620] Lustre: srv-fs26182-OST0001: No data found on store. Initialize space. [11046.353347] Lustre: 309110:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11046.450179] Lustre: fs26182-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11052.802721] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11062.374487] Lustre: server umount fs26182-OST0000 complete [11073.007705] LustreError: 308918:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11073.024140] LustreError: 308918:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [11073.321604] Lustre: server umount fs26182-OST0001 complete [11087.991544] loop1: detected capacity change from 0 to 204800000 [11094.388612] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11094.593637] Lustre: fs26182-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11101.090396] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11107.719951] loop2: detected capacity change from 0 to 204800000 [11113.320932] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11119.992120] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11127.780706] Lustre: server umount fs26182-OST0000 complete [11137.914401] Lustre: server umount fs26182-OST0001 complete [11149.793329] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11149.801123] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11149.813657] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11149.849614] Lustre: Skipped 1 previous similar message [11154.921812] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11154.930668] Lustre: Skipped 2 previous similar messages [11155.889055] Lustre: server umount lustre-OST0000 complete [11160.035166] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11160.038740] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11160.044510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11160.070239] Lustre: Skipped 1 previous similar message [11165.152922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11165.164024] Lustre: Skipped 2 previous similar messages [11165.975676] Lustre: server umount lustre-MDT0000 complete [11169.770978] LustreError: 301763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787708305 with bad export cookie 6492969811732808456 [11169.772611] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11169.782413] LustreError: 301763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11170.288558] Lustre: server umount lustre-MDT0001 complete [11182.465296] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [11185.551419] Key type lgssc unregistered [11185.891848] LNet: 315037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11185.898322] LNetError: 315037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11185.908726] LNet: Removed LNI 192.168.203.144@tcp [11187.005153] Key type .llcrypt unregistered [11187.014754] Key type ._llcrypt unregistered [11220.991904] Key type ._llcrypt registered [11220.995420] Key type .llcrypt registered [11221.188099] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [11238.806447] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [11240.264600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11240.342855] alg: No test for adler32 (adler32-zlib) [11241.444603] Lustre: Lustre: Build Version: 2.17.57_80_g4b309f5 [11241.862818] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [11243.592212] Key type lgssc registered [11244.878286] Lustre: Echo OBD driver; http://www.lustre.org/ [11282.522708] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11282.546348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11283.793194] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11283.822920] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11283.928747] Lustre: lustre-MDT0000: new disk, initializing [11284.006924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11284.021211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11288.659958] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11299.385859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11299.543558] Lustre: 318895:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11299.615431] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11299.624096] Lustre: Skipped 1 previous similar message [11299.743250] Lustre: lustre-MDT0001: new disk, initializing [11299.871909] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11299.941700] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11299.963682] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11304.622970] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11313.576569] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11318.356304] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11324.511163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11324.784818] Lustre: lustre-OST0000: new disk, initializing [11324.788053] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11324.799779] Lustre: 320869:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11324.896461] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11330.235321] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11331.628961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11331.639392] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11331.752391] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11337.760842] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11339.640466] Lustre: Mounted lustre-client - version 2.17.57_80_g4b309f5 [11340.597089] Lustre: Unmounted lustre-client [11349.990793] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11350.000365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11355.120782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11355.127371] Lustre: Skipped 1 previous similar message [11360.228793] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11360.236746] Lustre: Skipped 1 previous similar message [11362.272121] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11362.390957] Lustre: server umount lustre-OST0000 complete [11366.371530] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11366.379074] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11366.396831] Lustre: Skipped 1 previous similar message [11366.401911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11366.406257] Lustre: Skipped 1 previous similar message [11370.465489] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11370.471498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11370.501319] Lustre: Skipped 1 previous similar message [11380.193116] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11380.338034] Lustre: server umount lustre-MDT0000 complete [11383.686220] LustreError: 318889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787708519 with bad export cookie 2465460353002957993 [11383.691503] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11383.696254] LustreError: 318889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11383.996447] Lustre: server umount lustre-MDT0001 complete [11394.690788] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [11397.745069] Key type lgssc unregistered [11398.129709] LNet: 323035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11398.141587] LNetError: 323035:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11469.862914] LNet: Removed LNI 192.168.203.144@tcp [11470.582755] Key type .llcrypt unregistered [11470.585832] Key type ._llcrypt unregistered [11486.232300] Key type ._llcrypt registered [11486.235438] Key type .llcrypt registered [11486.343989] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [11501.902492] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [11502.710518] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11502.802103] alg: No test for adler32 (adler32-zlib) [11503.825770] Lustre: Lustre: Build Version: 2.17.57_80_g4b309f5 [11504.238734] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [11506.040624] Key type lgssc registered [11507.083225] Lustre: Echo OBD driver; http://www.lustre.org/ [11537.271487] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 21:44:30 (1787708670) [11544.421483] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11544.447321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11545.768298] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11545.798245] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11545.940870] Lustre: lustre-MDT0000: new disk, initializing [11546.079646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11546.102345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11550.886379] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11561.198146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11561.351475] Lustre: 326253:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11561.414692] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11561.421852] Lustre: Skipped 1 previous similar message [11561.494676] Lustre: lustre-MDT0001: new disk, initializing [11561.595793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11561.673692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11561.692421] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11565.866174] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11573.854127] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11578.780022] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11584.182882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11584.365610] Lustre: lustre-OST0000: new disk, initializing [11584.369361] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11584.374426] Lustre: 328224:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11584.439193] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11589.286151] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11590.718469] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11590.736192] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11590.829654] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11598.326410] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11611.627676] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11611.632532] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11611.633362] LustreError: Skipped 1 previous similar message [11611.634924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11611.651074] Lustre: Skipped 1 previous similar message [11611.665519] Lustre: Skipped 1 previous similar message [11616.742609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11616.753151] Lustre: Skipped 1 previous similar message [11617.750315] Lustre: server umount lustre-OST0000 complete [11621.861579] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11621.887929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11621.898463] Lustre: Skipped 1 previous similar message [11626.979267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11626.985281] Lustre: Skipped 1 previous similar message [11627.837245] Lustre: server umount lustre-MDT0000 complete [11631.580166] LustreError: 326246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787708767 with bad export cookie 11026921393492665960 [11631.588085] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11631.596085] LustreError: 326246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11631.942310] Lustre: server umount lustre-MDT0001 complete [11642.417204] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [11644.972615] Key type lgssc unregistered [11645.285303] LNet: 330132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11645.290373] LNetError: 330132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11645.317874] LNet: Removed LNI 192.168.203.144@tcp [11646.163400] Key type .llcrypt unregistered [11646.180098] Key type ._llcrypt unregistered [11665.839965] Key type ._llcrypt registered [11665.842643] Key type .llcrypt registered [11665.952640] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [11666.758370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11666.783539] alg: No test for adler32 (adler32-zlib) [11667.870560] Lustre: Lustre: Build Version: 2.17.57_80_g4b309f5 [11668.131872] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [11669.792231] Key type lgssc registered [11670.862745] Lustre: Echo OBD driver; http://www.lustre.org/ [11677.309977] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11677.324570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11678.796146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11682.974766] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11688.931608] LustreError: 331408:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11688.956484] LustreError: 331408:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11690.608708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11690.951784] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11694.806505] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11701.619989] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11707.362783] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11713.100738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11713.376307] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11718.944472] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11725.677619] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11734.310895] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11735.759725] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11742.178850] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11742.195468] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11742.209747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11744.227045] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11744.227405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11744.254270] Lustre: Skipped 1 previous similar message [11744.268063] Lustre: Skipped 3 previous similar messages [11745.490717] Lustre: server umount lustre-MDT0000 complete [11747.612024] LustreError: 332462:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11748.898877] LustreError: 331389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787708884 with bad export cookie 9441308932703354877 [11748.915398] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11748.919965] LustreError: 331389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11749.345568] LustreError: 331404:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11749.354832] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11749.395036] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11752.731238] LustreError: 331404:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11752.756565] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.44@tcp (stopping) [11755.314238] Lustre: server umount lustre-MDT0001 complete [11759.264740] Lustre: server umount lustre-OST0000 complete [11767.511119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11773.031767] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11776.387848] Lustre: 335584:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.144@tcp [11778.548547] Lustre: server umount MGS complete [11787.528604] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11788.106623] LustreError: 336156:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11788.138895] LustreError: 336156:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11788.236716] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11793.234375] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11797.795851] LustreError: 336151:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11797.833266] LustreError: 336151:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11802.656452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11802.901659] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.44@tcp (not set up) [11802.911199] Lustre: Skipped 1 previous similar message [11803.137347] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11807.769524] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11815.952193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11816.239692] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11822.808791] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11829.979429] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11831.355590] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11836.404993] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11839.975650] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [11848.161200] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11848.171911] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11848.185161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11849.192874] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11851.209475] Lustre: server umount lustre-OST0000 complete [11859.430955] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11859.433286] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11859.438024] Lustre: Skipped 1 previous similar message [11861.111846] Lustre: server umount lustre-MDT0000 complete [11864.545438] LustreError: 336152:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11864.566562] LustreError: 336152:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11864.826180] LustreError: 336136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787709000 with bad export cookie 9441308932703355717 [11864.835055] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11864.840075] LustreError: 336136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11865.222723] Lustre: server umount lustre-MDT0001 complete [11875.600796] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [11878.437604] Key type lgssc unregistered [11878.781540] LNet: 339595:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11878.800642] LNetError: 339595:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11878.819742] LNet: Removed LNI 192.168.203.144@tcp [11879.727189] Key type .llcrypt unregistered [11879.732290] Key type ._llcrypt unregistered [11898.395980] Key type ._llcrypt registered [11898.404482] Key type .llcrypt registered [11898.532634] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [11915.674397] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [11917.179354] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11917.199203] alg: No test for adler32 (adler32-zlib) [11918.731417] Lustre: Lustre: Build Version: 2.17.57_80_g4b309f5 [11919.088609] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [11920.858357] Key type lgssc registered [11922.623215] Lustre: Echo OBD driver; http://www.lustre.org/ [11959.926666] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11959.960711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11961.240115] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11961.297546] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11961.393245] Lustre: lustre-MDT0000: new disk, initializing [11961.463308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11961.478552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11965.815701] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11976.126414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11976.269796] Lustre: 342771:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11976.326597] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11976.331281] Lustre: Skipped 1 previous similar message [11976.401506] Lustre: lustre-MDT0001: new disk, initializing [11976.506360] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11976.559838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11976.570587] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11980.717241] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11989.989280] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11995.590802] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12002.417734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12002.751897] Lustre: lustre-OST0000: new disk, initializing [12002.758906] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12002.765736] Lustre: 344741:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12002.891719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12009.548622] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12010.096092] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12010.120975] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12010.278741] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12020.325838] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12026.069545] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12026.285256] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12032.463150] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12032.697583] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12036.074101] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12036.076937] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12036.079947] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12036.089619] LustreError: Skipped 1 previous similar message [12036.112929] Lustre: Skipped 1 previous similar message [12041.043971] Lustre: server umount lustre-OST0000 complete [12046.309745] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12046.330043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12046.335599] Lustre: Skipped 1 previous similar message [12046.361930] Lustre: Skipped 2 previous similar messages [12051.318670] Lustre: server umount lustre-MDT0000 complete [12055.369694] LustreError: 342763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787709191 with bad export cookie 16530036338579090343 [12055.371860] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12055.393448] LustreError: 342763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12055.747408] Lustre: server umount lustre-MDT0001 complete [12063.881537] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 21:53:17 (1787709197) [12079.326429] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [12088.667582] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [12121.646475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12121.846759] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12121.909396] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12121.991097] Lustre: lustre-MDT0000: new disk, initializing [12122.073828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12122.107732] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12125.928669] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12136.158693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12136.260881] Lustre: 349515:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12136.312517] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12136.316077] Lustre: Skipped 1 previous similar message [12136.371110] Lustre: lustre-MDT0001: new disk, initializing [12136.442340] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12136.479404] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12136.503627] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12140.778785] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12148.926760] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12154.166073] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12160.420211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12160.665528] Lustre: lustre-OST0000: new disk, initializing [12160.670952] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12160.678607] Lustre: 351482:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12160.766282] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12167.131861] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12168.749540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12168.769922] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12168.852487] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12176.203845] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12214.767026] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12214.783650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12214.792905] Lustre: Skipped 1 previous similar message [12216.989428] Lustre: server umount lustre-OST0000 complete [12223.457209] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12223.475846] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12223.504614] Lustre: Skipped 1 previous similar message [12223.512776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12227.263995] Lustre: server umount lustre-MDT0000 complete [12230.118284] LustreError: 352302:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12230.135175] LustreError: 352302:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12230.655264] LustreError: 351481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787709366 with bad export cookie 16530036338579091225 [12230.661865] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12230.664766] LustreError: 351481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12231.017752] Lustre: server umount lustre-MDT0001 complete [12242.252624] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12245.029804] Key type lgssc unregistered [12245.291122] LNet: 353658:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12245.296322] LNetError: 353658:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12246.313531] LNet: Removed LNI 192.168.203.144@tcp [12247.103755] Key type .llcrypt unregistered [12247.105333] Key type ._llcrypt unregistered [12331.173856] Key type ._llcrypt registered [12331.176807] Key type .llcrypt registered [12331.289859] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12331.572413] Key type .llcrypt unregistered [12331.574631] Key type ._llcrypt unregistered [12406.950252] Key type ._llcrypt registered [12406.953435] Key type .llcrypt registered [12407.069620] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12407.409581] Key type .llcrypt unregistered [12407.413299] Key type ._llcrypt unregistered [12433.873794] Key type ._llcrypt registered [12433.876511] Key type .llcrypt registered [12433.997338] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12434.445259] Key type .llcrypt unregistered [12434.448421] Key type ._llcrypt unregistered [12464.209845] Key type ._llcrypt registered [12464.213943] Key type .llcrypt registered [12464.328980] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12464.802926] Key type .llcrypt unregistered [12464.806908] Key type ._llcrypt unregistered [12543.105775] Key type ._llcrypt registered [12543.108454] Key type .llcrypt registered [12543.204890] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12543.592476] Key type .llcrypt unregistered [12543.594205] Key type ._llcrypt unregistered [12573.867928] Key type ._llcrypt registered [12573.873441] Key type .llcrypt registered [12574.000370] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12574.462414] Key type .llcrypt unregistered [12574.466236] Key type ._llcrypt unregistered [12607.592518] Key type ._llcrypt registered [12607.596899] Key type .llcrypt registered [12607.710324] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12608.058535] Key type .llcrypt unregistered [12608.067148] Key type ._llcrypt unregistered [12638.570613] Key type ._llcrypt registered [12638.574533] Key type .llcrypt registered [12638.661431] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12638.949465] Key type .llcrypt unregistered [12638.951343] Key type ._llcrypt unregistered [12669.161351] Key type ._llcrypt registered [12669.166960] Key type .llcrypt registered [12669.291289] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12669.637504] Key type .llcrypt unregistered [12669.641801] Key type ._llcrypt unregistered [12698.801996] Key type ._llcrypt registered [12698.805432] Key type .llcrypt registered [12698.920556] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12699.358394] Key type .llcrypt unregistered [12699.362392] Key type ._llcrypt unregistered [12760.905218] Key type ._llcrypt registered [12760.906367] Key type .llcrypt registered [12761.008305] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12761.272610] Key type .llcrypt unregistered [12761.282965] Key type ._llcrypt unregistered [12786.293196] Key type ._llcrypt registered [12786.295153] Key type .llcrypt registered [12786.418610] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12786.743419] Key type .llcrypt unregistered [12786.746531] Key type ._llcrypt unregistered [12812.615623] Key type ._llcrypt registered [12812.617773] Key type .llcrypt registered [12812.733761] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12812.999634] Key type .llcrypt unregistered [12813.002486] Key type ._llcrypt unregistered [12863.448737] Key type ._llcrypt registered [12863.452502] Key type .llcrypt registered [12863.571091] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [12879.062439] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [12880.281584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12880.325194] alg: No test for adler32 (adler32-zlib) [12881.312888] Lustre: Lustre: Build Version: 2.17.57_80_g4b309f5 [12881.570317] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [12883.216375] Key type lgssc registered [12884.279695] Lustre: Echo OBD driver; http://www.lustre.org/ [12917.749150] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12917.774471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12919.096496] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [12919.129830] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [12919.239756] Lustre: tst250fs-MDT0000: new disk, initializing [12919.342863] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12919.374147] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12924.134786] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12933.437807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12933.589569] Lustre: 373966:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12933.696510] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [12933.702946] Lustre: Skipped 1 previous similar message [12933.771816] Lustre: tst250fs-MDT0001: new disk, initializing [12933.867461] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12933.904075] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12933.912989] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12938.247298] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12942.905955] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12948.702618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12949.026416] Lustre: tst250fs-OST0000: new disk, initializing [12949.031831] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [12949.047342] Lustre: 375596:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12949.177923] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12951.674574] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12951.693596] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12951.847889] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12955.863167] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12968.199325] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12968.474372] Lustre: tst250fs-OST0001: new disk, initializing [12968.481392] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [12968.491697] Lustre: 376471:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12968.645755] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12975.752073] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12975.776390] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12975.922863] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12976.142683] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12982.832127] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [12985.589760] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [12985.600225] Lustre: Skipped 2 previous similar messages [13005.793539] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13005.798820] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13005.824989] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13006.830245] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13006.857076] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13006.873439] Lustre: Skipped 2 previous similar messages [13011.950944] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13011.966131] Lustre: Skipped 6 previous similar messages [13017.061684] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13017.067767] Lustre: Skipped 3 previous similar messages [13018.084807] Lustre: tst250fs-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13018.299958] Lustre: server umount tst250fs-MDT0000 complete [13021.436323] LustreError: 373959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787710157 with bad export cookie 3739844418037655508 [13021.440500] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13021.448074] LustreError: 373959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13021.870209] Lustre: server umount tst250fs-MDT0001 complete [13035.775815] Lustre: server umount tst250fs-OST0000 complete [13039.073496] Lustre: 372191:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787710158/real 1787710158] req@ffff96f10d2bdf80 x1874549818855808/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787710174 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13039.111750] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13039.466058] Lustre: server umount tst250fs-OST0001 complete [13058.522592] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [13067.506817] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [13103.754727] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 22:10:36 (1787710236) [13105.469721] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13123.627512] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [13135.048971] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [13170.609347] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12877 sec ========= 22:11:43 (1787710303) [13172.509673] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 22:11:45 (1787710305) === [13176.416900] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 22:11:49 (1787710309) === [13219.847851] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [13222.895560] Key type lgssc unregistered [13223.233471] LNet: 385604:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13223.237602] LNetError: 385604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13223.264523] LNet: Removed LNI 192.168.203.144@tcp [13224.297152] Key type .llcrypt unregistered [13224.301062] Key type ._llcrypt unregistered