[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 487848560 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.002412] x2apic enabled [ 0.003011] Switched APIC routing to physical x2apic. [ 0.004023] kvm-guest: setup PV IPIs [ 0.007448] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008033] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010012] pid_max: default: 32768 minimum: 301 [ 0.011153] LSM: Security Framework initializing [ 0.012077] Yama: becoming mindful. [ 0.013084] SELinux: Initializing. [ 0.014091] *** VALIDATE selinux *** [ 0.022838] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026862] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027208] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028135] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029159] *** VALIDATE tmpfs *** [ 0.030544] *** VALIDATE proc *** [ 0.031357] *** VALIDATE cgroup *** [ 0.032019] *** VALIDATE cgroup2 *** [ 0.033318] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034190] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035015] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036040] Spectre V2 : User space: Vulnerable [ 0.037013] Speculative Store Bypass: Vulnerable [ 0.040216] debug: unmapping init [mem 0xffffffff8d859000-0xffffffff8d860fff] [ 0.042208] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043759] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044036] ... version: 2 [ 0.045019] ... bit width: 48 [ 0.046018] ... generic registers: 4 [ 0.047016] ... value mask: 0000ffffffffffff [ 0.048027] ... max period: 00007fffffffffff [ 0.049022] ... fixed-purpose events: 3 [ 0.050015] ... event mask: 000000070000000f [ 0.051355] rcu: Hierarchical SRCU implementation. [ 0.053683] smp: Bringing up secondary CPUs ... [ 0.054659] x86: Booting SMP configuration: [ 0.055044] .... node #0, CPUs: #1 #2 #3 [ 0.059013] smp: Brought up 1 node, 4 CPUs [ 0.061026] smpboot: Max logical packages: 1 [ 0.062019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.227019] node 0 deferred pages initialised in 163ms [ 0.230293] devtmpfs: initialized [ 0.231288] x86/mm: Memory block size: 128MB [ 0.233797] gcov: version magic: 0x41383552 [ 0.235646] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.239089] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.243459] pinctrl core: initialized pinctrl subsystem [ 0.245186] [ 0.246009] ************************************************************* [ 0.249015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.252025] ** ** [ 0.254011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.257021] ** ** [ 0.259023] ** This means that this kernel is built to expose internal ** [ 0.262019] ** IOMMU data structures, which may compromise security on ** [ 0.265033] ** your system. ** [ 0.268029] ** ** [ 0.271017] ** If you see this message and you are not debugging the ** [ 0.274085] ** kernel, report this immediately to your vendor! ** [ 0.276017] ** ** [ 0.279012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.282017] ************************************************************* [ 0.285597] NET: Registered protocol family 16 [ 0.288452] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.291062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.293066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.297132] cpuidle: using governor menu [ 0.298642] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.299667] PCI: Using configuration type 1 for base access [ 0.300167] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.307130] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.310095] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.315048] cryptd: max_cpu_qlen set to 1000 [ 0.318317] ACPI: Added _OSI(Module Device) [ 0.320020] ACPI: Added _OSI(Processor Device) [ 0.322017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.324018] ACPI: Added _OSI(Processor Aggregator Device) [ 0.329015] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.335652] ACPI: Interpreter enabled [ 0.336084] ACPI: PM: (supports S0 S3 S4 S5) [ 0.338018] ACPI: Using IOAPIC for interrupt routing [ 0.340247] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.343452] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.354028] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.357063] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.360028] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.363118] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.368561] acpiphp: Slot [2] registered [ 0.370218] acpiphp: Slot [5] registered [ 0.372220] acpiphp: Slot [6] registered [ 0.373190] acpiphp: Slot [7] registered [ 0.375222] acpiphp: Slot [8] registered [ 0.377184] acpiphp: Slot [9] registered [ 0.378250] acpiphp: Slot [10] registered [ 0.380169] acpiphp: Slot [3] registered [ 0.381128] acpiphp: Slot [4] registered [ 0.383148] acpiphp: Slot [11] registered [ 0.385232] acpiphp: Slot [12] registered [ 0.386175] acpiphp: Slot [13] registered [ 0.387098] acpiphp: Slot [14] registered [ 0.389133] acpiphp: Slot [15] registered [ 0.391107] acpiphp: Slot [16] registered [ 0.392096] acpiphp: Slot [17] registered [ 0.393125] acpiphp: Slot [18] registered [ 0.395193] acpiphp: Slot [19] registered [ 0.396109] acpiphp: Slot [20] registered [ 0.397105] acpiphp: Slot [21] registered [ 0.398170] acpiphp: Slot [22] registered [ 0.400101] acpiphp: Slot [23] registered [ 0.401127] acpiphp: Slot [24] registered [ 0.403114] acpiphp: Slot [25] registered [ 0.404102] acpiphp: Slot [26] registered [ 0.406091] acpiphp: Slot [27] registered [ 0.407110] acpiphp: Slot [28] registered [ 0.409114] acpiphp: Slot [29] registered [ 0.410154] acpiphp: Slot [30] registered [ 0.412140] acpiphp: Slot [31] registered [ 0.413100] PCI host bridge to bus 0000:00 [ 0.415033] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.417024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.420023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.422026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.425027] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.427025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.429181] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.432092] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.435368] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.446746] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.452624] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.455021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.457020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.460020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.463591] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.466803] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.469051] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.471939] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.477018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.489954] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.495024] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.505032] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.521018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.537017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.559022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.573829] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.583016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.595018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.618023] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.628421] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.633017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.637016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.655025] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.668714] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.676015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.684015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.708018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.721000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.730019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.737020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.757019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.767759] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.775016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.782016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.800017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.811987] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.814464] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.817479] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.819403] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.824699] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.830034] iommu: Default domain type: Passthrough [ 0.831603] SCSI subsystem initialized [ 0.833125] ACPI: bus type USB registered [ 0.835108] usbcore: registered new interface driver usbfs [ 0.836071] usbcore: registered new interface driver hub [ 0.838107] usbcore: registered new device driver usb [ 0.840215] pps_core: LinuxPPS API ver. 1 registered [ 0.842012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.847132] PTP clock support registered [ 0.849191] EDAC MC: Ver: 3.0.0 [ 0.851179] PCI: Using ACPI for IRQ routing [ 0.854051] NetLabel: Initializing [ 0.855019] NetLabel: domain hash size = 128 [ 0.857022] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.860106] NetLabel: unlabeled traffic allowed by default [ 0.863169] vgaarb: loaded [ 0.865323] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.867016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.873816] clocksource: Switched to clocksource kvm-clock [ 0.996495] VFS: Disk quotas dquot_6.6.0 [ 0.998283] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.000859] *** VALIDATE ramfs *** [ 1.002211] *** VALIDATE hugetlbfs *** [ 1.003731] pnp: PnP ACPI init [ 1.006116] pnp: PnP ACPI: found 6 devices [ 1.028874] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.032229] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.034289] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.036677] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.039288] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.041602] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.044504] NET: Registered protocol family 2 [ 1.047279] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.053170] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.056662] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.062916] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.066241] TCP: Hash tables configured (established 65536 bind 65536) [ 1.069510] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.072770] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.076038] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.078600] NET: Registered protocol family 1 [ 1.080690] RPC: Registered named UNIX socket transport module. [ 1.085118] RPC: Registered udp transport module. [ 1.087268] RPC: Registered tcp transport module. [ 1.089131] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.092312] NET: Registered protocol family 44 [ 1.093957] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.096202] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.098460] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.100715] PCI: CLS 0 bytes, default 64 [ 1.102249] Unpacking initramfs... [ 2.529958] debug: unmapping init [mem 0xffff9e663cc54000-0xffff9e663ffbffff] [ 2.535168] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.537751] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.540552] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.052694] Initialise system trusted keyrings [ 3.056059] Key type blacklist registered [ 3.058335] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.068037] zbud: loaded [ 3.071401] *** VALIDATE nfs *** [ 3.072961] *** VALIDATE nfs4 *** [ 3.074741] pstore: using deflate compression [ 3.078358] Platform Keyring initialized [ 3.195239] NET: Registered protocol family 38 [ 3.197179] Key type asymmetric registered [ 3.198566] Asymmetric key parser 'x509' registered [ 3.200519] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.203956] io scheduler mq-deadline registered [ 3.205778] io scheduler kyber registered [ 3.207351] io scheduler bfq registered [ 3.209290] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.212705] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.215775] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.219182] ACPI: Power Button [PWRF] [ 3.225191] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.232857] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.250911] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.259615] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.278395] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.306611] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.336324] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.341982] Non-volatile memory driver v1.3 [ 3.343852] Linux agpgart interface v0.103 [ 3.381992] virtio_blk virtio1: [vda] 149488 512-byte logical blocks (76.5 MB/73.0 MiB) [ 3.385332] vda: detected capacity change from 0 to 76537856 [ 3.401264] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.403957] vdb: detected capacity change from 0 to 1073741824 [ 3.419613] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.422791] vdc: detected capacity change from 0 to 2621440000 [ 3.438715] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.443075] vdd: detected capacity change from 0 to 2621440000 [ 3.457072] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.459555] vde: detected capacity change from 0 to 4294967296 [ 3.476508] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.480143] vdf: detected capacity change from 0 to 4294967296 [ 3.490291] libphy: Fixed MDIO Bus: probed [ 3.495243] usbcore: registered new interface driver usbserial_generic [ 3.497925] usbserial: USB Serial support registered for generic [ 3.500464] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.505095] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.507457] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.510622] mousedev: PS/2 mouse device common for all mice [ 3.513899] rtc_cmos 00:05: RTC can wake from S4 [ 3.516860] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.521142] rtc_cmos 00:05: registered as rtc0 [ 3.523271] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.526829] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.527303] intel_pstate: CPU model not supported [ 3.532729] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.537627] hid: raw HID events driver (C) Jiri Kosina [ 3.539864] usbcore: registered new interface driver usbhid [ 3.541985] usbhid: USB HID core driver [ 3.543689] drop_monitor: Initializing network drop monitor service [ 3.546195] Initializing XFRM netlink socket [ 3.548424] NET: Registered protocol family 10 [ 3.551811] Segment Routing with IPv6 [ 3.553195] NET: Registered protocol family 17 [ 3.555427] mpls_gso: MPLS GSO support [ 3.561738] RAS: Correctable Errors collector initialized. [ 3.563607] AVX version of gcm_enc/dec engaged. [ 3.565192] AES CTR mode by8 optimization enabled [ 3.652535] sched_clock: Marking stable (3652503893, 0)->(4638362809, -985858916) [ 3.657424] registered taskstats version 1 [ 3.660238] Loading compiled-in X.509 certificates [ 3.662896] zswap: loaded using pool lzo/zbud [ 3.693401] Key type big_key registered [ 3.707287] Key type encrypted registered [ 3.709625] ima: No TPM chip found, activating TPM-bypass! [ 3.711984] ima: Allocated hash algorithm: sha1 [ 3.713871] ima: No architecture policies found [ 3.715705] evm: Initialising EVM extended attributes: [ 3.717490] evm: security.selinux [ 3.718794] evm: security.ima [ 3.719907] evm: security.capability [ 3.721321] evm: HMAC attrs: 0x1 [ 3.723701] rtc_cmos 00:05: setting system clock to 2026-08-22 10:25:00 UTC (1787394300) [ 3.730373] debug: unmapping init [mem 0xffffffff8e803000-0xffffffff8e9fffff] [ 3.733515] debug: unmapping init [mem 0xffffffff8d582000-0xffffffff8d858fff] [ 3.743810] Write protecting the kernel read-only data: 28672k [ 3.747335] debug: unmapping init [mem 0xffffffff8bc03000-0xffffffff8bdfffff] [ 3.750562] debug: unmapping init [mem 0xffffffff8c514000-0xffffffff8c5fffff] [ 3.786739] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.795557] systemd[1]: Detected virtualization kvm. [ 3.797737] systemd[1]: Detected architecture x86-64. [ 3.799873] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.823621] systemd[1]: No hostname configured. [ 3.825623] systemd[1]: Set hostname to . [ 3.827567] random: systemd: uninitialized urandom read (16 bytes read) [ 3.830414] systemd[1]: Initializing machine ID from random generator. [ 3.887914] random: ln: uninitialized urandom read (6 bytes read) [ 3.974292] random: systemd: uninitialized urandom read (16 bytes read) [ 3.978337] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.983816] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.991531] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.645312] device-mapper: uevent: version 1.0.3 [ 4.648430] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.422122] virtio_net virtio0 ens2: renamed from eth0 [ 5.437878] random: fast init done [ 5.515966] scsi host0: ata_piix [ 5.545169] scsi host1: ata_piix [ 5.546781] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.549419] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.260266] dracut-initqueue[581]: RTNETLINK answers: File exists [ 10.226309] random: crng init done [ 10.229963] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.733538] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.939800] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.218800] SELinux: Disabled at runtime. [ 12.278212] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.287857] systemd[1]: Detected virtualization kvm. [ 12.290251] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.833997] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.837654] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.842462] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.848348] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.851866] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.859877] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.865839] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd Root File System. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached targ[ 13.069504] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS et Paths. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.423477] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.801724] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.876821] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.996656] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.022564] EDAC sbridge: Ver: 1.1.2 [ 15.706302] Key type dns_resolver registered [ 16.020832] NFS: Registering the id_resolver key type [ 16.022874] Key type id_resolver registered [ 16.024500] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg617-server login: [ 56.238136] libcfs: loading out-of-tree module taints kernel. [ 56.339692] Key type ._llcrypt registered [ 56.345534] Key type .llcrypt registered [ 56.519433] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [ 77.297886] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 79.826555] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 79.850196] alg: No test for adler32 (adler32-zlib) [ 81.621642] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [ 82.486370] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 84.239579] Key type lgssc registered [ 86.493582] Lustre: Echo OBD driver; http://www.lustre.org/ [ 102.290080] hrtimer: interrupt took 3067714 ns [ 103.206563] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 146.058661] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 161.328642] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 161.364832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 162.685657] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 162.734519] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 162.809665] Lustre: lustre-MDT0000: new disk, initializing [ 162.897822] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 162.915314] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 167.431257] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 181.960963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 182.082780] Lustre: 6511:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 182.111864] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 182.117797] Lustre: Skipped 1 previous similar message [ 182.248371] Lustre: lustre-MDT0001: new disk, initializing [ 182.321985] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 182.357463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 182.376493] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 186.681969] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 191.246201] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 201.220519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 201.498911] Lustre: lustre-OST0000: new disk, initializing [ 201.504874] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 201.518449] Lustre: 8449:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 201.589068] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 201.711730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 201.724672] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 201.778552] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 208.036971] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 224.137949] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 224.304538] Lustre: lustre-OST0001: new disk, initializing [ 224.309921] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 224.317753] Lustre: 9523:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 224.418515] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 232.580761] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 234.043418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 234.051416] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 234.144557] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 245.781331] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 251.373831] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 258.848370] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing check_logdir /tmp/testlogs/ [ 265.237584] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing yml_node [ 272.654737] Lustre: DEBUG MARKER: Client: 2.17.54.115 [ 276.461813] Lustre: DEBUG MARKER: MDS: 2.17.54.115 [ 279.493293] Lustre: DEBUG MARKER: OSS: 2.17.54.115 [ 281.346571] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 22 06:29:35 EDT 2026 [ 299.346247] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 301.046821] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 320.480523] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 320.491577] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 320.507980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 321.508046] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 321.511461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 321.520802] Lustre: Skipped 1 previous similar message [ 321.529426] Lustre: Skipped 1 previous similar message [ 326.624664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 326.636802] Lustre: Skipped 3 previous similar messages [ 331.231157] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 331.438561] Lustre: server umount lustre-MDT0000 complete [ 334.665618] LustreError: 6504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394631 with bad export cookie 10775375587881547086 [ 334.674120] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 334.684521] LustreError: 6504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 336.863992] LustreError: 6518:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 336.866980] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 336.867926] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 336.880944] LustreError: 6518:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 336.892910] Lustre: Skipped 2 previous similar messages [ 339.875533] LustreError: 6516:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 339.899220] LustreError: 6516:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 341.236946] Lustre: server umount lustre-MDT0001 complete [ 345.024630] Lustre: server umount lustre-OST0000 complete [ 348.429066] Lustre: server umount lustre-OST0001 complete [ 355.672673] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [ 365.054067] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 399.338120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 399.633916] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 399.678810] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 399.744837] Lustre: lustre-MDT0000: new disk, initializing [ 399.860202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 399.881312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 403.885344] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 413.701339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 413.830222] Lustre: 15076:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 413.883921] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 413.886245] Lustre: Skipped 1 previous similar message [ 413.933933] Lustre: lustre-MDT0001: new disk, initializing [ 414.060870] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 414.109542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 414.118126] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 418.488905] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 427.708820] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 434.251916] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 441.590446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 441.918836] Lustre: lustre-OST0000: new disk, initializing [ 441.922861] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 441.929705] Lustre: 17044:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 442.043698] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 443.481590] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 443.488800] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 443.642477] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 448.927085] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 458.340092] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 463.150910] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 463.442473] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 468.403619] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 468.642178] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 474.602830] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 474.607979] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 474.644638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 474.657321] Lustre: Skipped 4 previous similar messages [ 476.868624] Lustre: server umount lustre-OST0000 complete [ 480.739088] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 480.752386] LustreError: Skipped 1 previous similar message [ 480.756856] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 480.764849] Lustre: Skipped 1 previous similar message [ 484.832684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 484.843989] Lustre: Skipped 2 previous similar messages [ 486.574812] Lustre: server umount lustre-MDT0000 complete [ 489.737867] LustreError: 18381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394786 with bad export cookie 10775375587881548619 [ 489.745058] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 489.747526] LustreError: 18381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 490.011783] Lustre: server umount lustre-MDT0001 complete [ 495.674486] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 06:33:10 (1787394790) [ 515.243485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 515.587470] LustreError: 19918:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 515.668660] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 520.648202] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 520.689727] LustreError: 19919:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 525.795412] LustreError: 19919:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 531.335673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 537.467213] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 546.976853] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 553.762663] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 556.485836] Lustre: Failing over lustre-MDT0000 [ 556.745828] Lustre: server umount lustre-MDT0000 complete [ 557.031661] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 557.032993] LustreError: 19914:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 557.063645] Lustre: Skipped 2 previous similar messages [ 557.085225] LustreError: 19914:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 572.386284] Lustre: 3646:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787394853/real 1787394853] req@ffff9e66b574b800 x1874218848520576/t0(0) o400->MGC192.168.206.117@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787394869 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 572.419857] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 576.828169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 577.504922] LustreError: 21031:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 577.539046] LustreError: 21031:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 582.908374] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 582.914386] Lustre: Skipped 1 previous similar message [ 588.076227] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 588.265053] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 588.292442] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 588.324789] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 599.453061] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 608.738590] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 608.741546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 608.742240] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 608.776038] Lustre: Skipped 1 previous similar message [ 608.800341] Lustre: Skipped 1 previous similar message [ 610.626789] Lustre: server umount lustre-MDT0000 complete [ 613.857310] LustreError: 19918:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 613.878725] LustreError: 19918:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 615.578525] LustreError: 19900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394912 with bad export cookie 10775375587881550033 [ 615.583675] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 615.589167] LustreError: 19900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 616.122844] Lustre: server umount lustre-MDT0001 complete [ 625.942911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 630.476385] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 639.293100] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 644.111971] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 652.179252] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 658.223592] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 660.209577] Lustre: Failing over lustre-MDT0000 [ 660.625943] Lustre: server umount lustre-MDT0000 complete [ 661.476642] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 677.861046] Lustre: 3644:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787394958/real 1787394958] req@ffff9e6582e7d880 x1874218848551168/t0(0) o400->MGC192.168.206.117@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787394974 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 677.892115] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 677.917235] LustreError: 24500:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 677.954949] LustreError: 24500:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 680.057658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 688.418657] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 692.954512] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 693.739340] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 693.752249] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 693.766961] Lustre: Skipped 1 previous similar message [ 693.796407] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 703.927554] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 709.088637] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 709.094239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 709.115207] Lustre: Skipped 1 previous similar message [ 713.931804] Lustre: server umount lustre-MDT0000 complete [ 718.253179] LustreError: 23238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395015 with bad export cookie 10775375587881550971 [ 718.257700] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 718.268642] LustreError: 23238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 718.747433] Lustre: server umount lustre-MDT0001 complete [ 727.177195] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 06:37:01 (1787395021) [ 744.303078] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [ 754.909525] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 795.755186] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 806.843447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 807.171391] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 807.219820] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 807.330522] Lustre: lustre-MDT0000: new disk, initializing [ 807.445287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 807.452282] Lustre: Skipped 3 previous similar messages [ 807.465143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 811.995708] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 823.258265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 823.363339] Lustre: 29632:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 823.756271] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 828.278719] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 838.524931] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 845.114592] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 853.418329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 853.832859] Lustre: 31601:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 855.592981] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 855.635708] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 855.689941] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 855.723396] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 861.303539] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 872.569759] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 879.128888] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 901.605347] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 901.615774] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 901.633859] Lustre: Skipped 3 previous similar messages [ 901.646355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 901.653058] Lustre: Skipped 2 previous similar messages [ 907.247169] LustreError: 31586:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 907.266464] LustreError: 31586:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 907.382112] Lustre: server umount lustre-OST0000 complete [ 918.894021] Lustre: server umount lustre-MDT0000 complete [ 923.129144] LustreError: 29626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395219 with bad export cookie 10775375587881551377 [ 923.135411] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 923.158647] LustreError: 29626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 923.830769] Lustre: server umount lustre-MDT0001 complete [ 937.697294] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 941.040451] Key type lgssc unregistered [ 941.405804] LNet: 33579:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 941.413660] LNetError: 33579:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 941.432826] LNet: Removed LNI 192.168.206.117@tcp [ 942.345156] Key type .llcrypt unregistered [ 942.346954] Key type ._llcrypt unregistered [ 971.092811] Key type ._llcrypt registered [ 971.100262] Key type .llcrypt registered [ 971.258712] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 973.139750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 973.246482] alg: No test for adler32 (adler32-zlib) [ 974.273706] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [ 974.526515] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 976.247390] Key type lgssc registered [ 977.698401] Lustre: Echo OBD driver; http://www.lustre.org/ [ 986.692113] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 986.718561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 988.220663] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 993.779677] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 997.347806] LustreError: 34994:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 997.377539] LustreError: 34994:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1002.481238] LustreError: 34993:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1002.677457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1003.133575] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1008.460942] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1017.552267] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1024.882737] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1032.729625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1033.296488] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1038.364342] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1040.337928] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1047.566468] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1048.548983] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1064.152836] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 06:42:38 (1787395358) [ 1066.537095] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1068.964757] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1077.212943] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 06:42:51 (1787395371) [ 1086.905945] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 06:43:00 (1787395380) [ 1088.101318] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1098.442864] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1098.450491] Lustre: Skipped 2 previous similar messages [ 1100.524135] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 06:43:14 (1787395394) [ 1102.911701] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1102.918189] Lustre: Skipped 2 previous similar messages [ 1112.258928] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1112.275776] Lustre: Skipped 4 previous similar messages [ 1129.086885] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 06:43:43 (1787395423) [ 1132.494874] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1136.617263] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1146.298264] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 06:44:00 (1787395440) [ 1148.601622] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1148.610478] Lustre: Skipped 6 previous similar messages [ 1164.021413] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 06:44:18 (1787395458) [ 1181.672562] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1181.676523] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1181.685170] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1185.763177] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1185.769993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1185.770888] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1187.103508] Lustre: server umount lustre-OST0000 complete [ 1192.417440] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1192.425775] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1192.454977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1192.459597] Lustre: Skipped 1 previous similar message [ 1196.004255] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1196.014160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1196.039968] Lustre: Skipped 1 previous similar message [ 1197.757314] Lustre: server umount lustre-MDT0000 complete [ 1201.125148] LustreError: 34994:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1201.160906] LustreError: 34994:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1201.665712] LustreError: 36840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395498 with bad export cookie 6526496520531335672 [ 1201.672657] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1201.684253] LustreError: 36840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1202.207993] Lustre: server umount lustre-MDT0001 complete [ 1213.381776] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 1216.135861] Key type lgssc unregistered [ 1216.471645] LNet: 42856:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1216.480494] LNetError: 42856:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1216.500563] LNet: Removed LNI 192.168.206.117@tcp [ 1217.483330] Key type .llcrypt unregistered [ 1217.486482] Key type ._llcrypt unregistered [ 1237.309170] Key type ._llcrypt registered [ 1237.314360] Key type .llcrypt registered [ 1237.431265] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [ 1261.287451] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 1263.855693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1263.905438] alg: No test for adler32 (adler32-zlib) [ 1265.424446] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [ 1265.905652] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 1267.719162] Key type lgssc registered [ 1269.348881] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1305.774960] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1305.814857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1307.185606] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1307.227236] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1307.319563] Lustre: lustre-MDT0000: new disk, initializing [ 1307.399623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1307.414227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1311.659982] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1321.955305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1322.083600] Lustre: 46014:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1322.146804] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1322.150597] Lustre: Skipped 1 previous similar message [ 1322.214798] Lustre: lustre-MDT0001: new disk, initializing [ 1322.306482] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1322.344874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1322.358757] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1326.773754] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1338.370344] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1344.034353] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1353.163751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1353.410063] Lustre: lustre-OST0000: new disk, initializing [ 1353.417681] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1353.430895] Lustre: 47983:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1353.554670] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1360.530555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1360.562716] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1360.792328] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1361.348479] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1371.227758] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1376.977427] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1377.175792] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1382.004104] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1382.254862] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1386.477825] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1386.482166] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1386.496281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1390.535367] Lustre: server umount lustre-OST0000 complete [ 1396.706996] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1396.718205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1396.729862] Lustre: Skipped 2 previous similar messages [ 1396.746968] Lustre: Skipped 2 previous similar messages [ 1400.770262] Lustre: server umount lustre-MDT0000 complete [ 1404.415200] LustreError: 46007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395701 with bad export cookie 14642738604208272029 [ 1404.417695] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1404.436224] LustreError: 46007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1404.766853] Lustre: server umount lustre-MDT0001 complete [ 1415.165770] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 06:48:29 (1787395709) [ 1421.875355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1422.278303] LustreError: 50442:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1422.313411] LustreError: 50442:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1422.382690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1427.116358] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1427.431238] LustreError: 50443:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1432.551254] LustreError: 50442:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1436.212189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1436.726085] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1441.717250] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1449.628766] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1456.170205] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1463.310116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1463.923094] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1470.664487] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1479.216132] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1486.942328] LustreError: 52971:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1488.079385] Lustre: Modifying parameter general.timeout=1 in log params [ 1489.098510] Lustre: Modifying parameter general.timeout=2 in log params [ 1491.617151] Lustre: Modifying parameter general.timeout=5 in log params [ 1491.619696] Lustre: Skipped 2 previous similar messages [ 1496.464973] Lustre: Modifying parameter general.timeout=10 in log params [ 1496.470837] Lustre: Skipped 4 previous similar messages [ 1504.981174] Lustre: Modifying parameter general.timeout=19 in log params [ 1504.987508] Lustre: Skipped 8 previous similar messages [ 1521.049956] Lustre: Modifying parameter general.timeout=36 in log params [ 1521.053606] Lustre: Skipped 16 previous similar messages [ 1553.544594] Lustre: Modifying parameter general.timeout=71 in log params [ 1553.549346] Lustre: Skipped 34 previous similar messages [ 1621.338553] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 06:51:55 (1787395915) [ 1622.371996] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_policies=tbf in log params [ 1622.377482] Lustre: Skipped 59 previous similar messages [ 1624.436761] Lustre: 59922:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1626.434139] Lustre: 60020:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1628.552414] Lustre: 60119:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1634.222539] Lustre: 60434:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1634.239358] Lustre: 60434:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1664.739674] Lustre: 61796:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1664.766071] LustreError: 61796:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1669.669295] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 06:52:43 (1787395963) [ 1681.890432] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1681.904948] Lustre: Skipped 2 previous similar messages [ 1681.906899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1681.913928] Lustre: Skipped 2 previous similar messages [ 1687.008804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1687.456195] Lustre: server umount lustre-MDT0000 complete [ 1692.121187] LustreError: 50423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395988 with bad export cookie 14642738604208272911 [ 1692.132717] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1692.137214] LustreError: 50437:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1692.137637] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1692.139256] LustreError: 50423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1692.145585] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1692.205599] Lustre: Skipped 2 previous similar messages [ 1692.645538] Lustre: server umount lustre-MDT0001 complete [ 1699.586902] Lustre: server umount lustre-OST0000 complete [ 1727.638382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1727.913838] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1728.194971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1732.458929] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1741.497791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1741.576893] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1741.611664] Lustre: 64326:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1746.303535] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1754.031356] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1759.972217] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1766.825873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1767.203091] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1767.445195] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1767.454973] Lustre: Skipped 1 previous similar message [ 1773.393410] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1781.046423] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1783.068299] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1783.079046] Lustre: Skipped 22 previous similar messages [ 1783.134301] Lustre: 66889:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1792.993639] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1793.015939] LustreError: Skipped 1 previous similar message [ 1793.025446] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1793.049158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1798.238331] Lustre: server umount lustre-OST0000 complete [ 1803.232304] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1803.238486] LustreError: Skipped 1 previous similar message [ 1803.247419] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1803.261069] Lustre: Skipped 1 previous similar message [ 1809.473662] Lustre: server umount lustre-MDT0000 complete [ 1813.507656] LustreError: 64318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396110 with bad export cookie 14642738604208275977 [ 1813.512978] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1813.520030] LustreError: 64318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1813.993246] Lustre: server umount lustre-MDT0001 complete [ 1825.657353] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 1828.453609] Key type lgssc unregistered [ 1828.750451] LNet: 68102:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1828.763511] LNetError: 68102:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1828.778768] LNet: Removed LNI 192.168.206.117@tcp [ 1829.684276] Key type .llcrypt unregistered [ 1829.686139] Key type ._llcrypt unregistered [ 1855.616522] Key type ._llcrypt registered [ 1855.618498] Key type .llcrypt registered [ 1855.692428] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 1856.658110] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1856.760847] alg: No test for adler32 (adler32-zlib) [ 1857.941855] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [ 1858.181252] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 1859.880478] Key type lgssc registered [ 1860.892187] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1868.444245] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1868.465861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1869.858293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1874.898851] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1880.039406] LustreError: 69515:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1880.058114] LustreError: 69515:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1882.972186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1883.527319] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1888.218027] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1895.423459] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1901.207380] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1908.487664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1908.974550] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1915.397528] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 1922.837289] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1930.410633] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1939.936478] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1939.947500] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1939.978943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1940.963142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1940.970190] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1940.984179] Lustre: Skipped 1 previous similar message [ 1941.008266] Lustre: Skipped 1 previous similar message [ 1946.082589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1946.085832] Lustre: Skipped 1 previous similar message [ 1946.299950] Lustre: server umount lustre-MDT0000 complete [ 1950.423130] LustreError: 69497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396247 with bad export cookie 3523748016883211730 [ 1950.434510] LustreError: 69497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1950.441540] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1950.921878] Lustre: server umount lustre-MDT0001 complete [ 1964.710506] Lustre: server umount lustre-OST0000 complete [ 1994.643219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1995.023410] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1995.071354] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1995.475154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1999.867633] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2009.483672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2009.618125] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2009.670065] Lustre: 74392:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2010.264731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2015.613600] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2023.012196] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2028.373873] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2034.866655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2035.175964] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2035.590137] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2042.478816] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2050.295306] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2052.147496] Lustre: 76969:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2054.064326] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2061.285023] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2061.306866] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2061.345742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2061.352054] Lustre: Skipped 2 previous similar messages [ 2065.483908] Lustre: server umount lustre-OST0000 complete [ 2071.520761] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2071.536684] LustreError: Skipped 1 previous similar message [ 2071.548781] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2071.583817] Lustre: Skipped 1 previous similar message [ 2071.594892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2071.606690] Lustre: Skipped 1 previous similar message [ 2075.854899] Lustre: server umount lustre-MDT0000 complete [ 2077.156228] LustreError: 74403:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2077.166283] LustreError: 74403:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2079.366428] LustreError: 74384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396376 with bad export cookie 3523748016883212570 [ 2079.375101] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2079.384825] LustreError: 74384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 2079.708972] Lustre: server umount lustre-MDT0001 complete [ 2089.842361] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 2092.891088] Key type lgssc unregistered [ 2093.287812] LNet: 78178:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2093.301863] LNetError: 78178:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2093.320870] LNet: Removed LNI 192.168.206.117@tcp [ 2094.238646] Key type .llcrypt unregistered [ 2094.243142] Key type ._llcrypt unregistered [ 2117.907450] Key type ._llcrypt registered [ 2117.912265] Key type .llcrypt registered [ 2118.070607] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 2119.364025] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2119.414401] alg: No test for adler32 (adler32-zlib) [ 2120.541749] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [ 2120.904017] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 2122.559493] Key type lgssc registered [ 2123.678109] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2132.127269] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2132.148395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2133.548632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2137.756714] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2143.721981] LustreError: 79594:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2143.738157] LustreError: 79594:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2146.390087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2147.105127] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2152.070348] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2160.612460] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2166.971070] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2175.729269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2176.570105] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2183.552656] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2192.252710] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2197.756379] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2199.184088] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2200.282058] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2202.554182] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2202.567422] Lustre: Skipped 1 previous similar message [ 2206.605430] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2206.609679] Lustre: Skipped 3 previous similar messages [ 2215.371996] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2215.377514] Lustre: Skipped 8 previous similar messages [ 2231.651977] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2231.659146] Lustre: Skipped 16 previous similar messages [ 2264.175947] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2264.182415] Lustre: Skipped 31 previous similar messages [ 2302.779941] Lustre: 87710:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2305.750728] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 07:03:19 (1787396599) [ 2307.691547] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2309.567648] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 07:03:23 (1787396603) [ 2314.727735] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2314.746761] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2314.772880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2314.787289] Lustre: Skipped 1 previous similar message [ 2319.246877] Lustre: server umount lustre-OST0000 complete [ 2324.963542] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2324.969894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2324.994484] Lustre: Skipped 2 previous similar messages [ 2330.082613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2330.091178] Lustre: Skipped 2 previous similar messages [ 2335.200836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2335.210719] Lustre: Skipped 1 previous similar message [ 2337.247120] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2337.544299] Lustre: server umount lustre-MDT0000 complete [ 2340.321738] LustreError: 79594:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2340.338574] LustreError: 79594:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2341.846132] LustreError: 80334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396638 with bad export cookie 2130712831084409388 [ 2341.855517] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2341.864411] LustreError: 80334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2342.684056] Lustre: server umount lustre-MDT0001 complete [ 2355.126138] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 2357.991948] Key type lgssc unregistered [ 2358.363862] LNet: 88964:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2358.373229] LNetError: 88964:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2358.396199] LNet: Removed LNI 192.168.206.117@tcp [ 2359.535262] Key type .llcrypt unregistered [ 2359.546362] Key type ._llcrypt unregistered [ 2364.062913] Key type ._llcrypt registered [ 2364.071549] Key type .llcrypt registered [ 2364.256564] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2368.970458] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules [ 2370.090950] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2370.221874] alg: No test for adler32 (adler32-zlib) [ 2371.282071] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [ 2371.589659] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 2373.320291] Key type lgssc registered [ 2374.631083] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2375.667905] LustreError: 90167:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2392.456093] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 2397.397597] LustreError: 90167:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2400.969711] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2400.996308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2402.613675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2406.946562] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2413.030472] LustreError: 91192:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2413.051694] LustreError: 91192:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2416.260901] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 07:05:10 (1787396710) [ 2418.155183] LustreError: 91191:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2421.719527] LustreError: 91223:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2421.733712] LustreError: 91223:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2421.747541] LustreError: 91223:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 2422.167493] Lustre: server umount lustre-MDT0000 complete [ 2437.236514] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 2440.371975] Key type lgssc unregistered [ 2440.675045] LNet: 92717:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2440.693803] LNetError: 92717:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2440.709452] LNet: Removed LNI 192.168.206.117@tcp [ 2441.557236] Key type .llcrypt unregistered [ 2441.561707] Key type ._llcrypt unregistered [ 2459.864636] Key type ._llcrypt registered [ 2459.866576] Key type .llcrypt registered [ 2459.943698] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 2460.750185] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2460.777214] alg: No test for adler32 (adler32-zlib) [ 2461.906830] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [ 2462.164422] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 2463.863255] Key type lgssc registered [ 2464.965837] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2473.169967] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2473.180958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2474.963542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2480.069554] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2485.218334] LustreError: 93934:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2485.242958] LustreError: 93934:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2490.336819] LustreError: 93935:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2490.900554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2491.868780] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2497.615458] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2504.936633] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2510.308561] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2517.461626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2518.100423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2524.279873] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2531.066399] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2583.942249] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 07:07:58 (1787396878) [ 2586.066356] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2588.191087] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 07:08:02 (1787396882) [ 2593.251458] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2593.263931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2593.272196] Lustre: Skipped 2 previous similar messages [ 2598.377959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2598.383785] Lustre: Skipped 2 previous similar messages [ 2598.992923] Lustre: server umount lustre-MDT0000 complete [ 2603.500629] LustreError: 93929:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2603.537356] LustreError: 93929:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2603.606191] LustreError: 93916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396900 with bad export cookie 6943557815802237983 [ 2603.614250] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2603.623469] LustreError: 93916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2603.889701] Lustre: server umount lustre-MDT0001 complete [ 2618.410820] Lustre: server umount lustre-OST0000 complete [ 2630.022746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2630.510538] LustreError: 98224:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2630.643701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2634.887789] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2635.748482] LustreError: 98225:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2643.623532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2644.231467] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2648.834933] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2656.486819] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2662.577548] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2676.706488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2676.876247] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2676.883225] LustreError: 98213:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2676.909175] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2676.916524] LustreError: 100307:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2676.921559] LustreError: 100307:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2676.925864] LustreError: 100307:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2676.993617] Lustre: server umount lustre-OST0000 complete [ 2676.998986] LustreError: 100307:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2686.886684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2687.079816] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2687.095872] LustreError: 98213:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2687.142428] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2687.162259] LustreError: 100773:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2687.184518] LustreError: 100773:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2687.193742] LustreError: 100773:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2687.286853] Lustre: server umount lustre-OST0000 complete [ 2687.291776] LustreError: 100773:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2696.958580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2697.103543] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2697.109679] Lustre: Found index 0 for lustre-OST0000, updating log [ 2697.113908] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2697.173936] Lustre: lustre-OST0000: new disk, initializing [ 2697.177298] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2697.182630] Lustre: 101309:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2697.482699] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2703.892603] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2707.293846] Lustre: Failing over lustre-OST0000 [ 2707.481823] Lustre: server umount lustre-OST0000 complete [ 2713.317364] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2713.320598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2713.334458] Lustre: Skipped 3 previous similar messages [ 2713.346955] Lustre: Skipped 1 previous similar message [ 2718.689485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2718.695842] Lustre: Skipped 1 previous similar message [ 2721.834668] Lustre: server umount lustre-MDT0000 complete [ 2723.809094] LustreError: 98225:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2723.829029] LustreError: 98225:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2725.436126] LustreError: 98206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397022 with bad export cookie 6943557815802240727 [ 2725.444512] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2725.450034] LustreError: 98206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2725.834907] Lustre: server umount lustre-MDT0001 complete [ 2734.299480] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 07:10:28 (1787397028) [ 2751.443791] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 2761.938299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2762.284735] LustreError: 103653:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2762.399811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2766.428079] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2775.089400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2781.578728] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2785.314500] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2792.776849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2793.457875] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2800.633289] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2800.652410] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2800.681752] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2800.774842] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2805.736573] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 2805.763434] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2807.778033] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:02 [ 2807.806287] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2807.831324] Lustre: lustre-OST0000: Recovery over after 0:07, of 1 clients 1 recovered and 0 were evicted. [ 2807.832914] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2807.855217] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2810.129053] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2810.265748] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2810.374475] Lustre: lustre-OST0001: new disk, initializing [ 2810.377904] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2810.389844] Lustre: 105975:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2810.801761] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2810.807136] Lustre: Skipped 2 previous similar messages [ 2816.407971] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2819.199235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2819.225755] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2819.314031] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2823.670488] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2833.488931] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2839.660499] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.17@tcp (stopping) [ 2840.038382] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2840.059650] Lustre: Skipped 2 previous similar messages [ 2849.852348] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.17@tcp (stopping) [ 2849.865644] Lustre: Skipped 7 previous similar messages [ 2852.319593] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2852.659699] Lustre: server umount lustre-MDT0001 complete [ 2854.974953] LustreError: 103647:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.206.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2854.990514] LustreError: 103647:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2864.175248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2864.345741] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2864.365808] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2864.502771] Lustre: 103642:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2866.579223] LustreError: 107543:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2866.595431] LustreError: 107543:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2866.629423] LustreError: 103644:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2866.809360] LustreError: 107554:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.206.117@tcp: cfg command failed: rc = -17 [ 2866.824402] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2866.824402] [ 2870.271479] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 2872.103787] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 2883.936107] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 07:12:57 (1787397177) [ 2903.686339] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 2923.613560] Lustre: 109590:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2946.672514] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2950.938560] Lustre: 110726:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3044.319832] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3044.321299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3044.346344] Lustre: Skipped 2 previous similar messages [ 3044.374819] Lustre: Skipped 6 previous similar messages [ 3049.377952] Lustre: server umount lustre-MDT0000 complete [ 3049.443425] LustreError: 104788:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3049.474919] LustreError: 104788:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 3053.161273] LustreError: 106694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397349 with bad export cookie 6943557815802241728 [ 3053.162117] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3053.171849] LustreError: 106694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3053.538716] Lustre: server umount lustre-MDT0001 complete [ 3068.054927] Lustre: server umount lustre-OST0000 complete [ 3070.947404] Lustre: 93246:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787397351/real 1787397351] req@ffff9e6690485880 x1874221343292544/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787397367 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3070.987110] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3071.005446] Lustre: Skipped 1 previous similar message [ 3071.877542] Lustre: server umount lustre-OST0001 complete [ 3077.330917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3087.541205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3095.293836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3104.897712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3126.611379] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 3137.012304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3137.047417] Lustre: lustre-MDT0000: reset Object Index mappings [ 3137.713048] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3137.721406] Lustre: Skipped 1 previous similar message [ 3142.480413] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3151.900707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3151.941051] Lustre: lustre-MDT0001: reset Object Index mappings [ 3157.896988] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3161.173576] Lustre: 115829:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3168.683576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3176.053678] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3177.580632] LustreError: 116182:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3177.589304] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:199 to 0x280000400:225) [ 3177.599173] LustreError: 116182:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3182.706147] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:382 to 0x280000401:418) [ 3185.155989] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3191.293603] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3191.312410] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3192.243193] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3199.636856] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3203.362953] Lustre: 117705:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3254.916302] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 07:19:09 (1787397549) [ 3283.445546] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3283.453045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3283.466169] Lustre: Skipped 1 previous similar message [ 3287.417189] Lustre: server umount lustre-MDT0000 complete [ 3292.048986] LustreError: 114661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397588 with bad export cookie 6943557815802288824 [ 3292.052267] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3292.058834] LustreError: 114661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3298.777301] Lustre: server umount lustre-MDT0001 complete [ 3308.927983] Lustre: server umount lustre-OST0000 complete [ 3319.001390] Lustre: server umount lustre-OST0001 complete [ 3325.427716] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [ 3335.065214] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 3368.576908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3368.753228] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3368.777643] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3368.842273] Lustre: lustre-MDT0000: new disk, initializing [ 3368.955127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3368.960044] Lustre: Skipped 3 previous similar messages [ 3368.976851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3372.921989] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3380.054363] Lustre: server umount lustre-MDT0000 complete [ 3390.414787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3390.736315] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3390.748993] Lustre: 121976:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3390.773271] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3395.592432] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3400.464546] Lustre: server umount lustre-MDT0000 complete [ 3409.342589] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 07:21:43 (1787397703) [ 3411.063719] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3412.968845] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 07:21:47 (1787397707) [ 3428.225978] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 3437.308929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3442.935871] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3452.357764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3452.494685] Lustre: 124026:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3452.592581] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3452.596929] Lustre: Skipped 1 previous similar message [ 3452.740637] Lustre: lustre-MDT0001: new disk, initializing [ 3452.909038] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3452.925754] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3457.869988] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3463.507965] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3470.665279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3471.016472] Lustre: lustre-OST0000: new disk, initializing [ 3471.019797] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3471.024670] Lustre: 125604:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3472.813234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3472.826458] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3472.969256] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3477.547769] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3486.978846] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3487.142126] Lustre: lustre-OST0001: new disk, initializing [ 3487.145886] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3487.151065] Lustre: 126474:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3488.896876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3488.908693] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3489.010232] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3492.703962] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 3501.768387] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3505.448321] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3878.040563] Lustre: 132893:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3878.048743] Lustre: 132893:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3878.060431] Lustre: 132893:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3878.067500] Lustre: 132893:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3878.078369] Lustre: 132893:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3878.083927] Lustre: 132893:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4180.601774] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 07:34:34 (1787398474) [ 4195.814779] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4195.815145] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4195.825948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4195.825956] Lustre: Skipped 7 previous similar messages [ 4195.866925] Lustre: Skipped 6 previous similar messages [ 4201.497928] Lustre: server umount lustre-MDT0000 complete [ 4205.189150] LustreError: 132905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787398501 with bad export cookie 6943557815802356969 [ 4205.194068] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4205.202095] LustreError: 132905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4206.050865] LustreError: 124032:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4206.054890] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4206.067985] LustreError: 124032:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 4206.094867] Lustre: Skipped 1 previous similar message [ 4206.101747] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4206.106191] Lustre: Skipped 8 previous similar messages [ 4211.671050] Lustre: server umount lustre-MDT0001 complete [ 4221.324817] Lustre: server umount lustre-OST0000 complete [ 4230.672633] Lustre: server umount lustre-OST0001 complete [ 4238.920668] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [ 4251.641885] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 4288.661228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4288.888283] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4288.912210] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4289.033270] Lustre: lustre-MDT0000: new disk, initializing [ 4289.100948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4289.107311] Lustre: Skipped 5 previous similar messages [ 4289.132847] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4293.714985] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 4303.146630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4303.259370] Lustre: 135952:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4303.309338] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4303.312453] Lustre: Skipped 1 previous similar message [ 4303.376648] Lustre: lustre-MDT0001: new disk, initializing [ 4303.494477] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4303.503513] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4307.460917] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 4316.664584] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4321.402634] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4327.829536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4328.156411] Lustre: lustre-OST0000: new disk, initializing [ 4328.161646] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4328.171679] Lustre: 137920:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4328.262057] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4328.271477] Lustre: Skipped 1 previous similar message [ 4330.176806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4330.184154] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4330.271956] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4333.807103] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 4343.426437] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4353.672191] Lustre: lustre-MDD0000: changelog on [ 4356.463570] Lustre: lustre-MDD0001: changelog on [ 6296.624525] Lustre: lustre-MDD0001: changelog off [ 6298.934111] Lustre: 135960:0:(llog_cat.c:984:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6300.272930] Lustre: lustre-MDD0000: changelog off [ 6309.886753] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6309.921334] Lustre: Skipped 1 previous similar message [ 6309.932484] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6309.941152] Lustre: Skipped 3 previous similar messages [ 6314.977972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6320.097129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6320.107656] Lustre: Skipped 1 previous similar message [ 6323.167177] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6323.405519] Lustre: server umount lustre-OST0000 complete [ 6330.338046] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6330.355264] Lustre: Skipped 1 previous similar message [ 6330.365239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6330.393620] Lustre: Skipped 1 previous similar message [ 6342.113270] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6342.467635] Lustre: server umount lustre-MDT0000 complete [ 6345.698090] LustreError: 137514:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6345.729069] LustreError: 137514:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6346.570077] LustreError: 135945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787400643 with bad export cookie 6943557815803177733 [ 6346.577567] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6346.581425] LustreError: 135945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6346.846634] Lustre: server umount lustre-MDT0001 complete [ 6360.383258] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 6363.552825] Key type lgssc unregistered [ 6363.902364] LNet: 147065:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6363.910808] LNetError: 147065:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6363.941590] LNet: Removed LNI 192.168.206.117@tcp [ 6364.964232] Key type .llcrypt unregistered [ 6364.966393] Key type ._llcrypt unregistered [ 6388.894942] Key type ._llcrypt registered [ 6388.902884] Key type .llcrypt registered [ 6389.048547] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [ 6406.338202] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 6407.468592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6407.500541] alg: No test for adler32 (adler32-zlib) [ 6408.626747] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [ 6408.902559] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 6410.567152] Key type lgssc registered [ 6412.045401] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6452.142321] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6452.171463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6453.489912] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6453.523302] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6453.583478] Lustre: lustre-MDT0000: new disk, initializing [ 6453.713888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6453.733257] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6458.881488] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6469.442489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6469.548567] Lustre: 150267:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6469.626659] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6469.631181] Lustre: Skipped 1 previous similar message [ 6469.703788] Lustre: lustre-MDT0001: new disk, initializing [ 6469.796138] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6469.862280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6469.881070] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6473.939923] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6483.365785] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6489.160679] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6495.667124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6495.922089] Lustre: lustre-OST0000: new disk, initializing [ 6495.925783] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6495.930111] Lustre: 152236:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6496.011342] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6503.316858] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6505.010463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6505.024278] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6505.153573] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6513.590230] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6520.005155] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_module obdecho/obdecho [ 6522.111011] LustreError: 153241:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6522.121826] LustreError: 153241:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6523.821195] LustreError: 153338:0:(echo_client.c:1883:echo_md_handler()) Can not resolve the path : rc = -2 [ 6541.283645] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6541.285897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6541.306916] Lustre: Skipped 2 previous similar messages [ 6541.320321] Lustre: Skipped 2 previous similar messages [ 6544.289766] Lustre: server umount lustre-MDT0000 complete [ 6546.403028] LustreError: 150276:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6546.412537] LustreError: 150276:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6547.799660] LustreError: 150259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787400844 with bad export cookie 8777906270378461625 [ 6547.806624] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6547.813120] LustreError: 150259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6548.167820] Lustre: server umount lustre-MDT0001 complete [ 6563.261665] Lustre: server umount lustre-OST0000 complete [ 6587.206578] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [ 6600.693495] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 6638.523993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6638.742748] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6638.762810] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6638.850353] Lustre: lustre-MDT0000: new disk, initializing [ 6638.979533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6639.038158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6643.521550] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6654.783229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6655.006815] Lustre: 157059:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6655.063843] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6655.068066] Lustre: Skipped 1 previous similar message [ 6655.145936] Lustre: lustre-MDT0001: new disk, initializing [ 6655.329055] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6655.400873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6655.421354] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6661.399324] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6672.908466] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6680.129558] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6687.683029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6687.978309] Lustre: lustre-OST0000: new disk, initializing [ 6687.985995] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6687.997851] Lustre: 159024:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6688.139505] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6694.572374] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6696.550571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6696.573677] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6696.738749] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6704.939420] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6710.146644] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6710.343141] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6715.442696] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6715.658585] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6722.528614] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6722.532744] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6722.540211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6722.563588] Lustre: Skipped 1 previous similar message [ 6722.576741] Lustre: Skipped 1 previous similar message [ 6727.649057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6727.667079] Lustre: Skipped 1 previous similar message [ 6732.255139] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6732.600584] Lustre: server umount lustre-OST0000 complete [ 6737.378087] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6737.384653] LustreError: Skipped 1 previous similar message [ 6737.395139] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6737.412931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6743.012389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6743.025110] Lustre: Skipped 2 previous similar messages [ 6743.898978] Lustre: server umount lustre-MDT0000 complete [ 6748.136852] LustreError: 157064:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6748.173630] LustreError: 157064:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6748.346618] LustreError: 159023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787401045 with bad export cookie 8777906270378462584 [ 6748.347907] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6748.356437] LustreError: 159023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6748.914673] Lustre: server umount lustre-MDT0001 complete [ 6752.722811] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 08:17:27 (1787401047) [ 6754.623049] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6756.769581] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 08:17:30 (1787401050) [ 6765.092853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6765.421453] LustreError: 161437:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6765.500315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6770.520308] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6770.666054] LustreError: 161438:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6775.778140] LustreError: 161437:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6779.790323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6785.246657] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6793.156570] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6799.549425] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6807.021912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6807.485177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6807.492094] Lustre: Skipped 1 previous similar message [ 6814.204484] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6822.312123] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6833.185891] LustreError: 161464:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x240000bd0:0x2:0x0]: rc = -108 [ 6833.217956] LustreError: 161464:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000006631163 x1874225481761792/t0(0) o1000->lustre-MDT0001-osp-MDT0000@0@lo:24/4 lens 264/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6833.253669] Lustre: 161464:0:(llog_cat.c:788:llog_cat_cancel_arr_rec()) lustre-MDT0001-osp-MDT0000: fail to cancel 1 records in [0x240000bd0:0x2:0x0]: rc = -108 [ 6833.265837] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.17@tcp (stopping) [ 6833.556122] Lustre: server umount lustre-MDT0000 complete [ 6835.178173] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6835.185943] LustreError: 163931:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6835.199097] Lustre: Skipped 2 previous similar messages [ 6835.222785] LustreError: 163931:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6837.327429] LustreError: 161419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787401134 with bad export cookie 8777906270378463466 [ 6837.335905] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6840.288333] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6840.298357] Lustre: Skipped 1 previous similar message [ 6843.949234] Lustre: server umount lustre-MDT0001 complete [ 6850.714331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6863.990918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6864.231855] LustreError: 165280:0:(llog.c:1655:llog_backup()) MGC192.168.206.117@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6864.245596] Lustre: 165280:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.117@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6867.938036] LustreError: 165285:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6867.965200] LustreError: 165285:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6902.760584] LustreError: 165287:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6902.777853] LustreError: 165287:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 29 previous similar messages [ 6904.896641] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.17@tcp (not set up) [ 6904.915809] Lustre: Skipped 4 previous similar messages [ 6925.279325] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6925.288966] Lustre: 165280:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.206.117@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6966.269396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6970.423870] LustreError: 165287:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6970.457548] LustreError: 165287:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 44 previous similar messages [ 6971.388336] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6971.719769] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6980.737368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6982.626349] LustreError: 148490:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000057715eb0 x1874225481780224/t0(0) o250->MGC192.168.206.117@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6983.077312] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -11 [ 6987.235707] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 6988.287415] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6994.193523] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6995.938974] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7001.205400] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7003.009788] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7024.096133] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7024.102903] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7024.108417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7024.135388] Lustre: Skipped 2 previous similar messages [ 7034.335798] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7034.519695] Lustre: server umount lustre-MDT0000 complete [ 7037.844681] LustreError: 161418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787401334 with bad export cookie 8777906270378469920 [ 7037.856199] LustreError: 161418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7037.858249] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7038.319878] Lustre: server umount lustre-MDT0001 complete [ 7052.449286] Lustre: server umount lustre-OST0000 complete [ 7072.942310] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [ 7083.002183] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 7121.584556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7121.837173] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7121.868310] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7121.972662] Lustre: lustre-MDT0000: new disk, initializing [ 7122.089759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7122.097738] Lustre: Skipped 1 previous similar message [ 7122.121711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7126.025233] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7135.764814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7135.849549] Lustre: 170388:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7136.099326] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7140.440784] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7149.624958] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7155.602279] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7162.113060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7162.460378] Lustre: lustre-OST0000: new disk, initializing [ 7162.464808] Lustre: Skipped 1 previous similar message [ 7162.472801] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7162.478603] Lustre: Skipped 2 previous similar messages [ 7162.487742] Lustre: 172358:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7169.361661] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7170.666841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7170.680272] Lustre: Skipped 1 previous similar message [ 7170.689263] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7170.775765] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7178.447262] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7182.768793] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7182.981773] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7187.781824] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7187.972605] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7191.525934] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7191.535085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7191.540516] Lustre: Skipped 1 previous similar message [ 7191.542470] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7191.544378] Lustre: Skipped 8 previous similar messages [ 7204.321863] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7204.523890] Lustre: server umount lustre-OST0000 complete [ 7222.241985] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7222.470536] Lustre: server umount lustre-MDT0000 complete [ 7226.652588] LustreError: 170382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787401523 with bad export cookie 8777906270378486748 [ 7226.663294] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7226.667909] LustreError: 170382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7227.084277] Lustre: server umount lustre-MDT0001 complete [ 7235.872120] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 08:25:30 (1787401530) [ 7243.251134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7243.839461] LustreError: 174770:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7243.866444] LustreError: 174770:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 7248.416588] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7257.911334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7262.544733] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7270.394324] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7276.368292] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7284.185512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7291.480209] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7299.674782] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7313.443516] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 08:26:47 (1787401607) [ 7318.496591] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7318.502625] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7318.527554] Lustre: Skipped 3 previous similar messages [ 7323.304912] Lustre: server umount lustre-OST0000 complete [ 7328.742769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7328.758151] Lustre: Skipped 14 previous similar messages [ 7334.546910] Lustre: server umount lustre-MDT0000 complete [ 7338.512416] LustreError: 174751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787401635 with bad export cookie 8777906270378487630 [ 7338.530458] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7338.534229] LustreError: 174751:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7338.992478] Lustre: server umount lustre-MDT0001 complete [ 7351.340513] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 7354.284432] Key type lgssc unregistered [ 7354.580549] LNet: 178500:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7354.602516] LNetError: 178500:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7355.625558] LNet: Removed LNI 192.168.206.117@tcp [ 7356.601437] Key type .llcrypt unregistered [ 7356.603919] Key type ._llcrypt unregistered [ 7378.018549] Key type ._llcrypt registered [ 7378.020707] Key type .llcrypt registered [ 7378.173367] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 7379.545534] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7379.745508] alg: No test for adler32 (adler32-zlib) [ 7380.820118] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [ 7381.127277] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 7382.832090] Key type lgssc registered [ 7383.837979] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7392.740787] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7392.758679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7408.415383] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7413.535600] LustreError: 179745:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.117@tcp: failed processing log, type 4: rc = -110 [ 7418.655428] LustreError: 179745:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7418.665328] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7418.676583] LustreError: MGC192.168.206.117@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7418.690069] LustreError: 179745:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7418.702843] LustreError: 179745:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7418.714644] LustreError: 179745:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7418.830101] Lustre: server umount lustre-OST0000 complete [ 7418.833943] LustreError: 179745:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7429.068193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7430.832965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7436.672198] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7440.372958] LustreError: 180244:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7440.396162] LustreError: 180244:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7445.480935] LustreError: 180245:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7447.731697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7448.185852] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7453.908892] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7463.107948] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7469.907427] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7477.889452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7478.043771] LustreError: 182061:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7478.052594] Lustre: 182061:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.206.117@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7478.214177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7485.877767] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7494.655934] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7497.706554] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7497.710423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7497.718643] Lustre: Skipped 2 previous similar messages [ 7497.743963] Lustre: Skipped 2 previous similar messages [ 7502.828467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7502.844760] Lustre: Skipped 2 previous similar messages [ 7503.744311] Lustre: server umount lustre-MDT0000 complete [ 7507.799323] LustreError: 180226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787401804 with bad export cookie 6653656372472759820 [ 7507.802188] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7507.817126] LustreError: 180226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7507.940892] LustreError: 180240:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7507.941873] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7508.001478] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7508.265677] Lustre: server umount lustre-MDT0001 complete [ 7515.627227] Lustre: server umount lustre-OST0000 complete [ 7527.457570] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [ 7537.495967] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 7578.341975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7578.631926] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7578.665407] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7578.747764] Lustre: lustre-MDT0000: new disk, initializing [ 7578.846186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7578.861276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7583.876086] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7595.247523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7595.422835] Lustre: 185586:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7595.479885] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7595.495699] Lustre: Skipped 1 previous similar message [ 7595.576640] Lustre: lustre-MDT0001: new disk, initializing [ 7595.702570] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7595.746217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7595.761488] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7600.897685] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7613.435818] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7620.928522] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7630.003962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7630.467047] Lustre: lustre-OST0000: new disk, initializing [ 7630.480400] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7630.492271] Lustre: 187555:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7630.692849] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7638.154118] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7638.169215] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7638.400477] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7640.870731] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7652.419195] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7657.157503] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7657.390308] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7662.041247] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7662.315863] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7669.223711] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7669.229955] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7669.268792] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7669.274422] Lustre: Skipped 1 previous similar message [ 7670.933537] Lustre: server umount lustre-OST0000 complete [ 7677.925952] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7677.937679] LustreError: Skipped 1 previous similar message [ 7677.941407] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7677.954324] Lustre: Skipped 1 previous similar message [ 7677.960596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7681.971118] Lustre: server umount lustre-MDT0000 complete [ 7684.585758] LustreError: 187565:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7684.606791] LustreError: 187565:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7687.107323] LustreError: 185579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787401983 with bad export cookie 6653656372472760534 [ 7687.118149] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7687.131970] LustreError: 185579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7687.563128] Lustre: server umount lustre-MDT0001 complete [ 7698.953569] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 08:33:13 (1787401993) [ 7720.601728] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 7723.750893] Key type lgssc unregistered [ 7724.218861] LNet: 190504:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7724.230106] LNetError: 190504:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7724.257787] LNet: Removed LNI 192.168.206.117@tcp [ 7725.082588] Key type .llcrypt unregistered [ 7725.086890] Key type ._llcrypt unregistered [ 7744.736794] Key type ._llcrypt registered [ 7744.740585] Key type .llcrypt registered [ 7744.829176] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 7745.978879] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7746.299949] alg: No test for adler32 (adler32-zlib) [ 7747.325926] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [ 7747.549319] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 7749.201183] Key type lgssc registered [ 7750.446152] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7758.651361] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7758.680782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7760.223696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7765.414873] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7769.572311] LustreError: 191725:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7769.613839] LustreError: 191725:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7774.294424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7774.721721] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7779.928148] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7787.939548] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7794.019506] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7800.944615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7801.148820] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7801.158390] LustreError: 193591:0:(llog.c:1696:llog_backup()) MGC192.168.206.117@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7801.175576] Lustre: 193591:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.206.117@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7801.341793] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7807.644603] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7815.126492] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7819.440618] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7819.448104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7819.457945] Lustre: Skipped 1 previous similar message [ 7819.480732] Lustre: Skipped 2 previous similar messages [ 7823.628031] Lustre: server umount lustre-MDT0000 complete [ 7824.868363] LustreError: 191726:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7824.889024] LustreError: 191726:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7827.397188] LustreError: 191705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787402124 with bad export cookie 7340727283189806907 [ 7827.412921] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7827.635319] LustreError: 192456:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000006d03a73e x1874226885716096/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7827.802800] Lustre: server umount lustre-MDT0001 complete [ 7841.479523] Lustre: server umount lustre-OST0000 complete [ 7852.732905] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [ 7865.396249] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 7903.858252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7904.254170] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7904.288314] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7904.358786] Lustre: lustre-MDT0000: new disk, initializing [ 7904.435033] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7904.459989] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7909.789580] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7921.623732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7921.837933] Lustre: 197113:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7921.922930] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7921.925456] Lustre: Skipped 1 previous similar message [ 7921.930861] LustreError: 197896:0:(llog.c:1696:llog_backup()) MGC192.168.206.117@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7921.947178] LustreError: 197896:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7921.949174] LustreError: 197124:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7921.953163] Lustre: 197896:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.117@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7921.996394] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7922.004618] Lustre: Skipped 1 previous similar message [ 7922.106109] Lustre: lustre-MDT0001: new disk, initializing [ 7922.197773] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7922.234560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7922.251191] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7926.727650] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7937.002548] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7942.845852] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7950.626685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7950.961375] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7950.963498] Lustre: Skipped 1 previous similar message [ 7950.971213] LustreError: 199058:0:(llog.c:1696:llog_backup()) MGC192.168.206.117@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7950.981569] LustreError: 199058:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7950.988234] Lustre: 199058:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.117@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7951.001742] Lustre: 199058:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7951.025853] Lustre: lustre-OST0000: new disk, initializing [ 7951.031739] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7951.044661] Lustre: 199082:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7951.157371] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7957.594897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7957.606172] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7957.760886] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7958.267215] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 7967.692995] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7972.473780] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7972.704834] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7978.790545] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7979.135629] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7983.589482] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7983.634621] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7983.635051] Lustre: Skipped 2 previous similar messages [ 7983.649474] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7983.680092] Lustre: Skipped 1 previous similar message [ 7988.713417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7988.724871] Lustre: Skipped 1 previous similar message [ 7993.831816] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7993.846788] Lustre: Skipped 1 previous similar message [ 7995.871142] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7996.460082] Lustre: server umount lustre-OST0000 complete [ 8004.068502] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8004.070394] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8004.071888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8004.081733] Lustre: Skipped 1 previous similar message [ 8006.824207] Lustre: server umount lustre-MDT0000 complete [ 8009.185352] LustreError: 197125:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8009.227039] LustreError: 197125:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8010.819333] LustreError: 197105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787402307 with bad export cookie 7340727283189807628 [ 8010.828950] LustreError: 197105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8010.831496] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8011.163714] Lustre: server umount lustre-MDT0001 complete [ 8020.617198] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 08:38:34 (1787402314) [ 8039.463940] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 8050.809360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8051.278209] LustreError: 202223:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8051.351248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8055.989287] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8061.926869] LustreError: 202223:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8061.941769] LustreError: 202223:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8064.799445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8070.177729] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8073.294347] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8080.093658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8086.864131] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8096.726494] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8096.953502] Lustre: lustre-OST0001: new disk, initializing [ 8096.958046] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8096.973822] Lustre: 204531:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8097.092285] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8097.099431] Lustre: Skipped 2 previous similar messages [ 8104.068287] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8106.571284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8106.581498] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8106.714561] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8115.520989] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8122.689765] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8124.679668] Lustre: DEBUG MARKER: ADD OST3 [ 8130.598425] loop: module loaded [ 8130.685240] loop0: detected capacity change from 0 to 204800000 [ 8130.990974] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8131.008891] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8131.028480] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8137.333769] loop0: detected capacity change from 0 to 204800000 [ 8138.386915] Lustre: DEBUG MARKER: STOP OST3 [ 8142.915614] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8143.222545] Lustre: lustre-OST0003: new disk, initializing [ 8143.235108] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8143.255323] Lustre: 206504:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8144.920710] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8145.045322] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8145.059199] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8145.099455] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8146.975438] LustreError: 206668:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8147.522732] Lustre: Failing over lustre-OST0003 [ 8147.760577] Lustre: server umount lustre-OST0003 complete [ 8149.626929] LustreError: 203709:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.206.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8149.657601] LustreError: 203709:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8150.093278] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8150.511995] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8150.513801] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8160.143256] Lustre: DEBUG MARKER: START OST3 again [ 8169.043578] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8169.756700] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8169.773832] Lustre: Skipped 1 previous similar message [ 8169.805744] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8170.045173] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8171.266085] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8171.274427] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8178.373833] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8201.133821] Lustre: Failing over lustre-OST0003 [ 8201.226449] Lustre: server umount lustre-OST0003 complete [ 8202.215196] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8202.215374] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8202.220930] LustreError: Skipped 1 previous similar message [ 8202.226028] LustreError: 203709:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8202.226039] LustreError: 203709:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 8202.304814] Lustre: Skipped 2 previous similar messages [ 8221.162645] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8221.169305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8221.188377] Lustre: Skipped 3 previous similar messages [ 8221.203102] Lustre: Skipped 4 previous similar messages [ 8225.723545] Lustre: server umount lustre-MDT0000 complete [ 8229.355165] LustreError: 202204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787402526 with bad export cookie 7340727283189808510 [ 8229.361077] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8229.376032] LustreError: 202204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8235.825196] Lustre: server umount lustre-MDT0001 complete [ 8239.771471] Lustre: server umount lustre-OST0000 complete [ 8243.394786] Lustre: server umount lustre-OST0001 complete [ 8249.740409] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [ 8257.781788] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 8295.736394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8295.936739] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8295.976796] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8296.058564] Lustre: lustre-MDT0000: new disk, initializing [ 8296.164639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8300.182935] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8309.476124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8309.599903] Lustre: 211987:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8309.834164] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8309.839585] Lustre: Skipped 1 previous similar message [ 8309.873762] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8313.920577] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8322.560457] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8328.249393] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8334.571884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8334.782738] Lustre: lustre-OST0000: new disk, initializing [ 8334.786325] Lustre: Skipped 1 previous similar message [ 8334.789732] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8334.793191] Lustre: Skipped 2 previous similar messages [ 8334.798372] Lustre: 213959:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8335.911694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8335.920658] Lustre: Skipped 1 previous similar message [ 8335.925216] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8336.027563] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8339.976245] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8348.327604] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8353.172940] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8353.349290] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8357.885094] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8358.164660] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8361.441179] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8361.447112] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8361.455974] Lustre: Skipped 2 previous similar messages [ 8361.459984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8361.463976] Lustre: Skipped 4 previous similar messages [ 8374.239227] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8374.417628] Lustre: server umount lustre-OST0000 complete [ 8381.409876] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8381.425686] LustreError: Skipped 1 previous similar message [ 8383.957622] Lustre: server umount lustre-MDT0000 complete [ 8386.939128] LustreError: 212782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787402683 with bad export cookie 7340727283189810806 [ 8386.950346] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8387.191667] Lustre: server umount lustre-MDT0001 complete [ 8390.273973] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 08:44:44 (1787402684) [ 8403.434546] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [ 8411.932435] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 8441.217384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8441.381665] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8441.400328] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8441.449039] Lustre: lustre-MDT0000: new disk, initializing [ 8441.531985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8445.535735] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8455.022964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8455.190954] Lustre: 218565:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8455.467220] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8459.711520] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8468.587238] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8474.481395] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8481.358099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8481.638738] Lustre: 220534:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8483.763191] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8487.429937] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8495.677924] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8499.842390] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8500.052955] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8504.652436] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8504.884418] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8509.408477] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8509.421168] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8509.422690] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8509.440559] Lustre: Skipped 4 previous similar messages [ 8509.460473] Lustre: Skipped 9 previous similar messages [ 8512.937317] Lustre: server umount lustre-OST0000 complete [ 8522.418300] Lustre: server umount lustre-MDT0000 complete [ 8524.770735] LustreError: 218578:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8524.782267] LustreError: 218578:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 18 previous similar messages [ 8525.502059] LustreError: 218560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787402822 with bad export cookie 7340727283189811688 [ 8525.512774] LustreError: 218560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8525.513404] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8525.824153] Lustre: server umount lustre-MDT0001 complete [ 8532.757366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8537.312378] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8544.653077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8549.013853] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8556.326702] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8561.876640] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8568.959761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8569.451249] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8569.462390] Lustre: Skipped 6 previous similar messages [ 8576.590780] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8585.707595] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8596.449370] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8596.471172] LustreError: Skipped 2 previous similar messages [ 8596.482868] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8596.491506] Lustre: Skipped 2 previous similar messages [ 8596.503663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8596.508804] Lustre: Skipped 3 previous similar messages [ 8602.418716] Lustre: server umount lustre-OST0000 complete [ 8612.871195] Lustre: server umount lustre-MDT0000 complete [ 8616.772368] LustreError: 223484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787402913 with bad export cookie 7340727283189812570 [ 8616.776543] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8616.783831] LustreError: 223484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8617.066638] Lustre: server umount lustre-MDT0001 complete [ 8724.483605] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 8726.814522] Key type lgssc unregistered [ 8727.115776] LNet: 226410:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8727.123424] LNetError: 226410:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8727.138716] LNet: Removed LNI 192.168.206.117@tcp [ 8727.830163] Key type .llcrypt unregistered [ 8727.836668] Key type ._llcrypt unregistered [ 8752.912632] Key type ._llcrypt registered [ 8752.914445] Key type .llcrypt registered [ 8753.032926] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [ 8769.879970] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 8771.004729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8771.025134] alg: No test for adler32 (adler32-zlib) [ 8772.051822] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [ 8772.266860] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 8773.887928] Key type lgssc registered [ 8775.383539] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8814.008514] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8814.030487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8815.296719] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8815.344400] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8815.438753] Lustre: lustre-MDT0000: new disk, initializing [ 8815.511425] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8815.532388] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8819.596687] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8829.922291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8830.029725] Lustre: 229764:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8830.079869] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8830.084281] Lustre: Skipped 1 previous similar message [ 8830.142498] Lustre: lustre-MDT0001: new disk, initializing [ 8830.219273] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8830.246482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8830.257735] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8834.252530] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8843.285920] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8848.825511] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8855.818527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8856.243276] Lustre: lustre-OST0000: new disk, initializing [ 8856.253312] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8856.264770] Lustre: 231732:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8856.392237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8862.456772] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8866.339344] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8866.357254] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8866.453691] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8871.461703] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8876.863462] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8877.138743] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8881.923631] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8882.152782] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8886.753965] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8886.762736] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8886.779515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8887.268182] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8887.280561] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8887.297050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8887.304707] Lustre: Skipped 1 previous similar message [ 8890.178273] Lustre: server umount lustre-OST0000 complete [ 8896.995309] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8897.003951] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8897.032666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8902.624279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8902.631287] Lustre: Skipped 3 previous similar messages [ 8907.754940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8907.765147] Lustre: Skipped 1 previous similar message [ 8908.255237] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8908.439031] Lustre: server umount lustre-MDT0000 complete [ 8912.736695] LustreError: 229757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787403209 with bad export cookie 14253720892208791359 [ 8912.747400] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8912.758111] LustreError: 229757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8913.025378] Lustre: server umount lustre-MDT0001 complete [ 8933.652083] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 8944.448590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8945.001105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8949.721483] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8950.248452] LustreError: 234667:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8950.284390] LustreError: 234667:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8954.349873] LustreError: 234668:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8959.465198] LustreError: 234667:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8959.909324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8960.506534] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8965.270480] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8968.560769] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8976.264147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8976.855673] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8984.156391] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 8994.266291] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8994.608916] Lustre: lustre-OST0001: new disk, initializing [ 8994.616980] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8994.626497] Lustre: 236975:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8994.783711] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9001.669494] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9002.123837] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9002.131496] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9002.255219] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9013.022286] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9026.039560] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9040.969124] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 08:55:35 (1787403335) [ 9043.428766] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9043.434476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9043.442681] Lustre: Skipped 3 previous similar messages [ 9043.466590] Lustre: Skipped 3 previous similar messages [ 9049.622622] Lustre: server umount lustre-MDT0000 complete [ 9053.674311] LustreError: 234664:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9053.693772] LustreError: 234664:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9053.909574] LustreError: 237744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787403350 with bad export cookie 14253720892208792234 [ 9053.921380] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9053.922776] LustreError: 237744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9054.261982] Lustre: server umount lustre-MDT0001 complete [ 9069.137565] Lustre: server umount lustre-OST0000 complete [ 9083.528665] Lustre: server umount lustre-OST0001 complete [ 9091.218888] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [ 9102.851613] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 9142.158685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9142.432534] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9142.463304] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9142.593970] Lustre: lustre-MDT0000: new disk, initializing [ 9142.709206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9142.740676] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9147.769744] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9159.890317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9160.066383] Lustre: 240927:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9160.185645] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9160.201472] Lustre: Skipped 1 previous similar message [ 9160.406986] Lustre: lustre-MDT0001: new disk, initializing [ 9160.628077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9160.641762] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9165.397372] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9175.494643] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9181.569291] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9190.377836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9190.929719] Lustre: lustre-OST0000: new disk, initializing [ 9190.940173] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9190.947768] Lustre: 242894:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9192.134043] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9192.146786] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9192.318855] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9198.968757] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9210.120874] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9215.144442] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9215.395804] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9220.252267] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9220.497838] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9223.138515] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9223.151838] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9223.165336] Lustre: Skipped 1 previous similar message [ 9223.170326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9223.177623] Lustre: Skipped 5 previous similar messages [ 9228.976494] Lustre: server umount lustre-OST0000 complete [ 9237.474877] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9237.484573] LustreError: Skipped 1 previous similar message [ 9237.490221] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9237.509071] Lustre: Skipped 1 previous similar message [ 9240.144832] Lustre: server umount lustre-MDT0000 complete [ 9243.620627] LustreError: 241802:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9243.630464] LustreError: 241802:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9244.260599] LustreError: 244231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787403541 with bad export cookie 14253720892208794376 [ 9244.281960] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9244.286822] LustreError: 244231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9244.894825] Lustre: server umount lustre-MDT0001 complete [ 9254.230134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9254.817761] LustreError: 245111:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9254.941701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9254.949245] Lustre: Skipped 2 previous similar messages [ 9259.594299] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9269.171364] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9275.325215] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9285.128469] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9291.604459] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9299.350928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9306.623431] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9314.171472] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9462.771160] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9462.794435] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9462.845152] Lustre: Skipped 2 previous similar messages [ 9462.849996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9462.885295] Lustre: Skipped 6 previous similar messages [ 9467.619970] Lustre: server umount lustre-OST0000 complete [ 9477.788891] Lustre: server umount lustre-MDT0000 complete [ 9478.115787] LustreError: 245111:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9478.126256] LustreError: 245111:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9481.345160] LustreError: 245093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787403778 with bad export cookie 14253720892208795258 [ 9481.358387] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9481.364121] LustreError: 245093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9481.768167] Lustre: server umount lustre-MDT0001 complete [ 9581.445844] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [ 9584.189170] Key type lgssc unregistered [ 9584.433026] LNet: 248733:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9584.440498] LNetError: 248733:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9584.465156] LNet: Removed LNI 192.168.206.117@tcp [ 9585.301219] Key type .llcrypt unregistered [ 9585.304347] Key type ._llcrypt unregistered [ 9614.103849] Key type ._llcrypt registered [ 9614.106662] Key type .llcrypt registered [ 9614.233600] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [ 9630.023766] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 9631.643849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9631.714028] alg: No test for adler32 (adler32-zlib) [ 9632.872437] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [ 9633.113564] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [ 9634.759220] Key type lgssc registered [ 9636.372536] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9682.348681] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 9692.643476] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9692.696309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9693.911278] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9693.938652] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9694.010701] Lustre: lustre-MDT0000: new disk, initializing [ 9694.073236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9694.097917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9698.570278] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9707.940978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9708.067580] Lustre: 252803:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9708.141595] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9708.146536] Lustre: Skipped 1 previous similar message [ 9708.269346] Lustre: lustre-MDT0001: new disk, initializing [ 9708.368855] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9708.396892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9708.403216] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9712.363917] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9716.913241] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9722.794083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9723.025917] Lustre: lustre-OST0000: new disk, initializing [ 9723.034492] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9723.051220] Lustre: 254433:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9723.137702] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9725.200241] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9725.213404] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9725.361620] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9728.413093] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9737.450913] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9737.628100] Lustre: lustre-OST0001: new disk, initializing [ 9737.632995] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9737.638385] Lustre: 255307:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9737.719115] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9743.294713] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9744.976811] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9744.987889] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9745.089472] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9753.208519] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9760.138208] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9770.980059] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9770.983577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9770.990963] Lustre: Skipped 3 previous similar messages [ 9770.998621] Lustre: Skipped 2 previous similar messages [ 9776.101552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9776.109942] Lustre: Skipped 3 previous similar messages [ 9776.801888] Lustre: server umount lustre-MDT0000 complete [ 9780.024334] LustreError: 252797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787404076 with bad export cookie 2445758765292887962 [ 9780.025726] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9780.031187] LustreError: 252797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9780.316176] Lustre: server umount lustre-MDT0001 complete [ 9793.763868] Lustre: server umount lustre-OST0000 complete [ 9797.087171] Lustre: 250308:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787404077/real 1787404077] req@ffff9e668899ad80 x1874228862330624/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787404093 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9797.117663] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9797.339761] Lustre: server umount lustre-OST0001 complete [ 9810.262949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9818.731168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9825.821179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9834.136873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9851.527344] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [ 9860.463889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9860.481358] Lustre: lustre-MDT0000: reset Object Index mappings [ 9860.874761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9864.512564] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9866.216363] LustreError: 260316:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9866.227589] LustreError: 260316:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9870.314152] LustreError: 260317:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9871.851827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9871.876507] Lustre: lustre-MDT0001: reset Object Index mappings [ 9872.190323] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9875.578266] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9877.917941] Lustre: 261457:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9883.550599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9885.931205] LustreError: 261809:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9885.953927] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9889.211388] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9891.305650] LustreError: 261809:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9896.429174] LustreError: 261809:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9896.458245] LustreError: 261809:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9897.351250] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9897.551744] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9897.560181] Lustre: Skipped 1 previous similar message [ 9902.591371] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9902.923342] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [ 9910.765867] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9914.079912] Lustre: 263327:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9917.930743] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9917.932947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9917.945041] Lustre: Skipped 2 previous similar messages [ 9917.955598] Lustre: Skipped 3 previous similar messages [ 9923.065170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9923.075297] Lustre: Skipped 3 previous similar messages [ 9924.016167] Lustre: server umount lustre-MDT0000 complete [ 9927.101898] LustreError: 262638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787404223 with bad export cookie 2445758765292894759 [ 9927.113693] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9927.119962] LustreError: 262638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9927.452812] Lustre: server umount lustre-MDT0001 complete [ 9941.254372] Lustre: server umount lustre-OST0000 complete [ 9943.522108] Lustre: 250308:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787404224/real 1787404224] req@ffff9e66ada2ed80 x1874228862377088/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787404240 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9943.550991] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9943.561615] Lustre: Skipped 1 previous similar message [ 9944.149397] Lustre: server umount lustre-OST0001 complete [ 9961.860228] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 09:10:56 (1787404256) [ 9976.918148] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [ 9987.695704] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [10022.209553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10022.419202] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10022.447225] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10022.524723] Lustre: lustre-MDT0000: new disk, initializing [10022.596781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10022.610388] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10026.815784] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10036.785786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10036.922366] Lustre: 267578:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10036.980773] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10036.984513] Lustre: Skipped 1 previous similar message [10037.121379] Lustre: lustre-MDT0001: new disk, initializing [10037.304884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10037.314551] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10042.022397] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10053.042941] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10061.121026] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10069.286912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10069.588716] Lustre: lustre-OST0000: new disk, initializing [10069.591840] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10069.599813] Lustre: 269549:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10071.431683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10071.441426] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10071.550465] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10075.476658] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10084.882547] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10089.171660] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10089.343880] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10094.193312] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10094.421912] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10097.120463] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10097.125187] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10097.139091] LustreError: Skipped 1 previous similar message [10097.140161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10097.156683] Lustre: Skipped 1 previous similar message [10097.184879] Lustre: Skipped 1 previous similar message [10102.637545] Lustre: server umount lustre-OST0000 complete [10107.362925] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10107.392842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10107.398845] Lustre: Skipped 3 previous similar messages [10112.456432] Lustre: server umount lustre-MDT0000 complete [10112.483561] LustreError: 269004:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10112.499107] LustreError: 269004:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10116.618611] LustreError: 267572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787404413 with bad export cookie 2445758765292896852 [10116.634796] LustreError: 267572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10116.635981] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10117.167628] Lustre: server umount lustre-MDT0001 complete [10135.709672] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [10144.890972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10145.325436] LustreError: 272482:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10145.375313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10145.381156] Lustre: Skipped 2 previous similar messages [10151.138768] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10159.893128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10164.501495] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10167.627819] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10173.353406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10179.086699] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10187.966617] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10188.206499] Lustre: lustre-OST0001: new disk, initializing [10188.210809] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10188.220723] Lustre: 274789:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10190.214070] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10190.225549] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10190.336150] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10193.997440] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10203.747194] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10207.333830] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10212.487451] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10212.548769] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10221.536488] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10221.549573] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10221.560717] Lustre: Skipped 1 previous similar message [10221.573958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10233.312798] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10233.562054] Lustre: server umount lustre-MDT0000 complete [10237.070539] LustreError: 274788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787404533 with bad export cookie 2445758765292897734 [10237.074181] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10237.079291] LustreError: 274788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10237.395860] Lustre: server umount lustre-MDT0001 complete [10251.258257] Lustre: server umount lustre-OST0000 complete [10253.791323] Lustre: 250309:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787404534/real 1787404534] req@ffff9e6581a85880 x1874228862488704/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787404550 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10253.842709] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10253.857690] Lustre: Skipped 3 previous similar messages [10255.330404] Lustre: server umount lustre-OST0001 complete [10273.573889] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [10284.258415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10284.646030] LustreError: 277846:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10284.677658] LustreError: 277846:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10284.718624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10284.727173] Lustre: Skipped 3 previous similar messages [10289.021846] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10296.781884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10301.941310] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10304.991504] Lustre: 278987:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10310.573446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10317.246028] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10326.076309] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10329.667693] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10329.680174] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10334.052364] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10342.020056] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10345.983290] Lustre: 280865:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10351.751538] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10352.037249] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10362.173728] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 09:17:36 (1787404656) [10379.237982] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10379.243552] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10379.257662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10379.262939] Lustre: Skipped 12 previous similar messages [10382.900389] Lustre: server umount lustre-MDT0000 complete [10385.893087] LustreError: 277842:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10385.915097] LustreError: 277842:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [10386.031607] LustreError: 277828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787404682 with bad export cookie 2445758765292904531 [10386.032633] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10386.048665] LustreError: 277828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10392.763794] Lustre: server umount lustre-MDT0001 complete [10403.111399] Lustre: server umount lustre-OST0000 complete [10407.500391] Lustre: server umount lustre-OST0001 complete [10414.924870] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [10425.319702] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [10470.683967] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [10480.241337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10480.492426] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10480.519177] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10480.634851] Lustre: lustre-MDT0000: new disk, initializing [10480.702328] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10485.073593] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10495.148045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10495.247750] Lustre: 285103:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10495.579589] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10499.638304] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10510.750564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10510.989145] Lustre: 286733:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10512.711722] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10517.334279] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10526.969920] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10527.136507] Lustre: 287607:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10529.066306] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10529.075453] Lustre: Skipped 1 previous similar message [10529.201264] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10533.311312] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10543.363713] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10546.729522] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10546.735657] Lustre: Skipped 1 previous similar message [10561.030506] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 09:20:55 (1787404855) [10565.095544] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10565.100169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10565.120251] Lustre: Skipped 8 previous similar messages [10565.142217] Lustre: Skipped 9 previous similar messages [10569.361411] Lustre: server umount lustre-MDT0000 complete [10570.211116] LustreError: 288440:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10570.228573] LustreError: 288440:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [10572.730554] LustreError: 285096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787404869 with bad export cookie 2445758765292911979 [10572.736832] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10572.745423] LustreError: 285096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10573.140862] Lustre: server umount lustre-MDT0001 complete [10587.386170] Lustre: server umount lustre-OST0000 complete [10600.605203] Lustre: server umount lustre-OST0001 complete [10608.831601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10609.321685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10609.328928] Lustre: Skipped 7 previous similar messages [10613.392949] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10621.053975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10625.109522] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10632.070667] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10637.172420] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10643.055370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10649.037939] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10657.569247] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10674.701198] loop1: detected capacity change from 0 to 204800000 [10674.945996] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10674.953053] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10674.964886] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10679.602916] loop1: detected capacity change from 0 to 204800000 [10685.407394] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10685.514676] LustreError: 289840:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs13893: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10685.526824] LustreError: 289840:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs13893 [10685.533591] LustreError: 289840:0:(mgs_handler.c:517:mgs_target_reg()) fs13893-MDT0001: Error checking target index: rc = -13 [10685.540796] LustreError: fs13893-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10685.549058] LustreError: 293571:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10685.553668] LustreError: 293571:0:(tgt_mount.c:2129:server_put_super()) no obd fs13893-MDT0001 [10685.557255] LustreError: 293571:0:(tgt_mount.c:116:server_deregister_mount()) fs13893-MDT0001 not registered [10685.596650] Lustre: server umount fs13893-MDT0001 complete [10685.599901] LustreError: 293571:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [10694.353937] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10694.433892] Lustre: Setting parameter fs13893-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs13893-MDT0001 [10694.479155] Lustre: srv-fs13893-MDT0001: No data found on store. Initialize space. [10694.490517] Lustre: Skipped 4 previous similar messages [10694.595904] Lustre: fs13893-MDT0001: new disk, initializing [10694.599084] Lustre: Skipped 3 previous similar messages [10694.714431] Lustre: srv-fs13893-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10694.730294] Lustre: 294039:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs13893-MDT0001-mdtlov: can't create new llog: rc = -115 [10694.746466] LustreError: 294039:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs13893-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10699.268809] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10706.260618] Lustre: server umount fs13893-MDT0001 complete [10719.513330] loop1: detected capacity change from 0 to 204800000 [10725.118764] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10725.473276] Lustre: 295605:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs13893-MDT0001-mdtlov: can't create new llog: rc = -115 [10725.486346] LustreError: 295605:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs13893-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10726.395226] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10729.394798] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10734.110322] Lustre: server umount fs13893-MDT0001 complete [10736.620615] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10744.294134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10749.010128] Lustre: server umount lustre-MDT0000 complete [10756.834458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10756.961420] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10761.400646] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10762.733791] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10768.921705] loop1: detected capacity change from 0 to 204800000 [10774.506796] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10775.227407] Lustre: 298051:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs13893-MDT0001-mdtlov: can't create new llog: rc = -115 [10775.240533] LustreError: 298051:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs13893-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10779.899575] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10784.672885] Lustre: server umount fs13893-MDT0001 complete [10808.292143] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10808.305921] LustreError: Skipped 1 previous similar message [10812.275714] Lustre: server umount lustre-OST0000 complete [10821.577075] Lustre: server umount lustre-MDT0000 complete [10824.677983] LustreError: 289832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787405121 with bad export cookie 2445758765292914660 [10824.692182] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10824.949796] Lustre: server umount lustre-MDT0001 complete [10833.917952] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [10836.311440] Key type lgssc unregistered [10836.580826] LNet: 300392:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10836.596913] LNetError: 300392:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10836.621233] LNet: Removed LNI 192.168.206.117@tcp [10837.502333] Key type .llcrypt unregistered [10837.510517] Key type ._llcrypt unregistered [10855.350124] Key type ._llcrypt registered [10855.352170] Key type .llcrypt registered [10855.447219] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [10856.385114] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10856.418482] alg: No test for adler32 (adler32-zlib) [10857.435618] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [10857.714175] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [10859.356531] Key type lgssc registered [10860.372767] Lustre: Echo OBD driver; http://www.lustre.org/ [10867.001544] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10867.015239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10868.427045] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10872.608994] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10878.945962] LustreError: 301666:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10878.958854] LustreError: 301666:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10879.658077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10879.950598] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10883.076447] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10889.026623] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10893.733588] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10899.602815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10900.029913] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10905.121470] LustreError: 303493:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10905.448285] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10910.184901] LustreError: 303821:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10912.419523] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10913.268276] LustreError: 303821:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10913.282021] LustreError: 303821:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10918.371858] LustreError: 303820:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10918.388837] LustreError: 303820:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10928.132763] loop1: detected capacity change from 0 to 204800000 [10928.262257] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10928.275978] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10928.287884] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10928.618315] LustreError: 303495:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10928.634108] LustreError: 303495:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [10932.270575] loop1: detected capacity change from 0 to 204800000 [10936.601317] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10936.749422] Lustre: fs3341-OST0000: new disk, initializing [10936.759238] Lustre: srv-fs3341-OST0000: No data found on store. Initialize space. [10936.766747] Lustre: 305399:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10936.857901] Lustre: fs3341-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10942.800217] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10946.110096] LustreError: 303493:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10946.140320] LustreError: 303493:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [10950.648972] Lustre: server umount fs3341-OST0000 complete [10962.969875] loop1: detected capacity change from 0 to 204800000 [10968.233620] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10968.526929] Lustre: fs3341-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10975.080496] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [10979.817135] LustreError: 303495:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10979.843535] LustreError: 303495:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 19 previous similar messages [10983.028782] loop2: detected capacity change from 0 to 204800000 [10983.236171] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10983.260764] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10983.277681] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10987.773154] loop2: detected capacity change from 0 to 204800000 [10990.083169] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10992.967683] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10993.018358] LustreError: fs3341-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10993.038426] LustreError: fs3341-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10993.058323] LustreError: 308441:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10993.066796] LustreError: 308441:0:(tgt_mount.c:2129:server_put_super()) no obd fs3341-OST0001 [10993.078633] LustreError: 308441:0:(tgt_mount.c:116:server_deregister_mount()) fs3341-OST0001 not registered [10993.158120] Lustre: server umount fs3341-OST0001 complete [10993.164056] LustreError: 308441:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [10995.176350] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11003.728985] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11003.974841] Lustre: fs3341-OST0001: new disk, initializing [11003.983410] Lustre: srv-fs3341-OST0001: No data found on store. Initialize space. [11003.995382] Lustre: 308992:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11004.131411] Lustre: fs3341-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11011.727454] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11020.881258] Lustre: server umount fs3341-OST0000 complete [11030.240481] Lustre: server umount fs3341-OST0001 complete [11041.828693] loop1: detected capacity change from 0 to 204800000 [11046.377968] LustreError: 303821:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11046.405632] LustreError: 303821:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 38 previous similar messages [11047.172878] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11047.378795] Lustre: fs3341-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11053.388648] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11059.845338] loop2: detected capacity change from 0 to 204800000 [11064.976486] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11071.030857] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11081.049709] Lustre: server umount fs3341-OST0000 complete [11090.045960] Lustre: server umount fs3341-OST0001 complete [11102.693163] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11102.697219] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11102.704870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11102.710965] Lustre: Skipped 1 previous similar message [11107.808566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11107.825592] Lustre: Skipped 2 previous similar messages [11108.129737] Lustre: server umount lustre-OST0000 complete [11112.930618] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11112.942746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11112.950031] Lustre: Skipped 1 previous similar message [11117.644506] Lustre: server umount lustre-MDT0000 complete [11121.168418] LustreError: 301648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787405417 with bad export cookie 4467789498190809959 [11121.171147] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11121.179058] LustreError: 301648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11121.725709] Lustre: server umount lustre-MDT0001 complete [11132.703021] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [11135.924317] Key type lgssc unregistered [11136.259898] LNet: 314918:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11136.271444] LNetError: 314918:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11136.289084] LNet: Removed LNI 192.168.206.117@tcp [11137.234635] Key type .llcrypt unregistered [11137.237793] Key type ._llcrypt unregistered [11165.642408] Key type ._llcrypt registered [11165.644192] Key type .llcrypt registered [11165.739123] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [11178.181847] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [11179.005480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11179.109263] alg: No test for adler32 (adler32-zlib) [11180.168943] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [11180.375182] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [11182.031171] Key type lgssc registered [11183.116721] Lustre: Echo OBD driver; http://www.lustre.org/ [11215.081893] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11215.112431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11216.349370] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11216.375125] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11216.444585] Lustre: lustre-MDT0000: new disk, initializing [11216.513279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11216.526907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11220.904614] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11231.171714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11231.280285] Lustre: 318756:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11231.342634] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11231.347268] Lustre: Skipped 1 previous similar message [11231.421574] Lustre: lustre-MDT0001: new disk, initializing [11231.505146] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11231.540405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11231.549070] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11236.175403] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11245.613404] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11251.175488] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11257.490464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11257.823313] Lustre: lustre-OST0000: new disk, initializing [11257.829836] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11257.839740] Lustre: 320727:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11257.921294] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11263.907444] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11267.684266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11267.697497] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11267.788939] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11272.947252] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11274.845364] Lustre: Mounted lustre-client [11275.738838] Lustre: Unmounted lustre-client [11284.965793] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11284.978870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11284.984488] Lustre: Skipped 1 previous similar message [11290.083037] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11290.095697] Lustre: Skipped 1 previous similar message [11290.465559] Lustre: server umount lustre-OST0000 complete [11295.201530] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11295.213791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11295.215986] Lustre: Skipped 2 previous similar messages [11299.827281] Lustre: server umount lustre-MDT0000 complete [11303.397614] LustreError: 320726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787405600 with bad export cookie 5003607600383682789 [11303.404976] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11303.415974] LustreError: 320726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11303.844992] Lustre: server umount lustre-MDT0001 complete [11313.481368] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [11316.046540] Key type lgssc unregistered [11336.863636] LNet: 322885:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11336.877778] LNetError: 322885:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11404.527198] LNet: Removed LNI 192.168.206.117@tcp [11405.533920] Key type .llcrypt unregistered [11405.536685] Key type ._llcrypt unregistered [11423.094892] Key type ._llcrypt registered [11423.097611] Key type .llcrypt registered [11423.202160] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [11438.472191] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [11439.656948] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11439.711955] alg: No test for adler32 (adler32-zlib) [11440.760520] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [11440.963747] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [11442.607295] Key type lgssc registered [11443.628815] Lustre: Echo OBD driver; http://www.lustre.org/ [11473.142811] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 09:36:07 (1787405767) [11479.612063] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11479.660935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11480.939162] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11480.968373] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11481.036427] Lustre: lustre-MDT0000: new disk, initializing [11481.101614] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11481.116643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11485.857267] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11495.990112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11496.129510] Lustre: 326111:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11496.225878] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11496.233301] Lustre: Skipped 1 previous similar message [11496.335363] Lustre: lustre-MDT0001: new disk, initializing [11496.442630] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11496.479880] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11496.495541] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11500.807116] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11509.108409] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11514.748631] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11521.088469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11521.347160] Lustre: lustre-OST0000: new disk, initializing [11521.353562] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11521.361753] Lustre: 328079:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11521.434842] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11527.204388] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11529.796513] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11529.809121] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11529.904184] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11536.064242] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11550.690684] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11550.693730] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11550.722307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11550.730890] Lustre: Skipped 1 previous similar message [11555.515500] Lustre: server umount lustre-OST0000 complete [11560.931569] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11560.935641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11560.943767] Lustre: Skipped 2 previous similar messages [11560.955654] Lustre: Skipped 1 previous similar message [11565.615206] Lustre: server umount lustre-MDT0000 complete [11569.083226] LustreError: 326914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787405865 with bad export cookie 10674530430502639523 [11569.088562] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11569.097967] LustreError: 326914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11569.478716] Lustre: server umount lustre-MDT0001 complete [11579.209962] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [11581.648355] Key type lgssc unregistered [11581.992924] LNet: 329986:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11582.001398] LNetError: 329986:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11582.017682] LNet: Removed LNI 192.168.206.117@tcp [11582.811632] Key type .llcrypt unregistered [11582.815364] Key type ._llcrypt unregistered [11601.536429] Key type ._llcrypt registered [11601.539887] Key type .llcrypt registered [11601.659809] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [11602.740152] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11602.771684] alg: No test for adler32 (adler32-zlib) [11603.821800] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [11604.183637] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [11605.872159] Key type lgssc registered [11607.196941] Lustre: Echo OBD driver; http://www.lustre.org/ [11614.766655] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11614.790595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11616.386845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11621.421613] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11626.997292] LustreError: 331269:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11627.022633] LustreError: 331269:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11630.630214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11631.012771] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11636.037893] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11643.187667] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11647.910298] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11654.719039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11655.075821] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11661.470143] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11669.023681] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11679.599597] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11681.222978] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11687.393754] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11687.400231] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11687.413151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11687.908035] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11687.918520] Lustre: Skipped 1 previous similar message [11687.923678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11687.926567] Lustre: Skipped 2 previous similar messages [11691.353236] Lustre: server umount lustre-MDT0000 complete [11692.598945] LustreError: 331264:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11695.722951] LustreError: 331251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787405992 with bad export cookie 8483505096725045295 [11695.732301] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11695.740562] LustreError: 331251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11697.755218] LustreError: 331265:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11697.804034] LustreError: 331265:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11697.823226] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.17@tcp (stopping) [11698.153500] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11700.193072] LustreError: 331266:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11700.195198] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11700.212415] LustreError: 331266:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11700.217098] Lustre: Skipped 1 previous similar message [11702.170235] Lustre: server umount lustre-MDT0001 complete [11706.563470] Lustre: server umount lustre-OST0000 complete [11717.062672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11722.987153] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11726.148545] Lustre: 335449:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.117@tcp [11728.220388] Lustre: server umount MGS complete [11737.152120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11737.579169] LustreError: 336021:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11737.676863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11742.877978] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11747.809884] LustreError: 336022:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11747.847512] LustreError: 336022:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11751.486453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11751.795684] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11756.241732] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11764.477189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11764.844839] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11771.354736] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11779.078953] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11781.003223] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11787.294940] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11789.104408] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11794.926654] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [11794.937720] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11794.962405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11796.058533] Lustre: server umount lustre-OST0000 complete [11803.104637] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11803.107394] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11803.110860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11803.116341] Lustre: Skipped 1 previous similar message [11813.343287] Lustre: 330581:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787406094/real 1787406094] req@00000000dc78238c x1874230929058688/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787406110 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11813.366800] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11815.391141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11815.682596] Lustre: server umount lustre-MDT0000 complete [11818.466373] Lustre: 330582:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787406099/real 1787406099] req@000000003f2471fc x1874230929060608/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787406115 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11818.467676] LustreError: 338077:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11818.524685] LustreError: 338077:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11820.565205] LustreError: 336003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787406117 with bad export cookie 8483505096725046135 [11820.570634] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11820.588100] LustreError: 336003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [11820.849495] Lustre: server umount lustre-MDT0001 complete [11834.959182] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [11838.240269] Key type lgssc unregistered [11838.524625] LNet: 339458:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11838.529946] LNetError: 339458:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11838.547809] LNet: Removed LNI 192.168.206.117@tcp [11839.518670] Key type .llcrypt unregistered [11839.521028] Key type ._llcrypt unregistered [11858.585476] Key type ._llcrypt registered [11858.587882] Key type .llcrypt registered [11858.691727] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [11873.507622] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [11874.939641] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11875.073389] alg: No test for adler32 (adler32-zlib) [11876.185934] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [11876.400992] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [11878.087221] Key type lgssc registered [11879.328552] Lustre: Echo OBD driver; http://www.lustre.org/ [11913.508363] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11913.528921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11914.783250] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11914.804529] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11914.890404] Lustre: lustre-MDT0000: new disk, initializing [11914.947846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11914.970361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11918.375027] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11927.995495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11928.134052] Lustre: 342619:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11928.180582] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11928.184203] Lustre: Skipped 1 previous similar message [11928.264237] Lustre: lustre-MDT0001: new disk, initializing [11928.362743] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11928.399757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11928.410736] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11932.446229] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11940.437786] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11944.708645] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11950.381918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11950.775972] Lustre: lustre-OST0000: new disk, initializing [11950.783571] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11950.791387] Lustre: 344587:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11950.868690] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11956.800873] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [11958.941245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11958.960417] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11959.130041] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11966.975682] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11972.705796] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11972.914888] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11977.029974] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11977.294775] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11979.744849] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11979.754664] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11979.769286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11979.786365] Lustre: Skipped 1 previous similar message [11984.882331] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11984.896492] Lustre: Skipped 2 previous similar messages [11985.809908] Lustre: server umount lustre-OST0000 complete [11989.984656] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11990.024024] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11990.039508] Lustre: Skipped 1 previous similar message [11995.104747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11995.114143] Lustre: Skipped 1 previous similar message [11995.934487] Lustre: server umount lustre-MDT0000 complete [11999.929486] LustreError: 342611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787406296 with bad export cookie 13214682286905799300 [11999.939405] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11999.945729] LustreError: 342611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12000.544956] Lustre: server umount lustre-MDT0001 complete [12008.167917] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 09:45:02 (1787406302) [12022.925435] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [12032.392051] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [12067.966793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12068.124346] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12068.172677] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12068.269859] Lustre: lustre-MDT0000: new disk, initializing [12068.346136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12068.370796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12072.262224] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [12082.742800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12082.932890] Lustre: 349343:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12082.995563] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12083.002866] Lustre: Skipped 1 previous similar message [12083.109540] Lustre: lustre-MDT0001: new disk, initializing [12083.201357] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12083.233618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12083.242865] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12087.333716] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [12096.891349] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12102.651185] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12109.716614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12110.147994] Lustre: lustre-OST0000: new disk, initializing [12110.151312] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12110.162965] Lustre: 351313:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12110.263919] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12116.083945] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [12117.531590] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12117.545145] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12117.647942] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12125.119756] Lustre: DEBUG MARKER: oleg617-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12158.440459] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12158.451391] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12158.457596] Lustre: Skipped 2 previous similar messages [12158.470805] Lustre: Skipped 1 previous similar message [12168.679157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12168.683385] Lustre: Skipped 3 previous similar messages [12171.231162] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12171.408732] Lustre: server umount lustre-OST0000 complete [12175.335939] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12175.347151] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12180.872378] Lustre: server umount lustre-MDT0000 complete [12184.241040] LustreError: 349336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787406481 with bad export cookie 13214682286905800182 [12184.254799] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12184.257395] LustreError: 349336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12184.643360] Lustre: server umount lustre-MDT0001 complete [12193.814103] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [12196.881966] Key type lgssc unregistered [12197.234845] LNet: 353487:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12197.240735] LNetError: 353487:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12197.261687] LNet: Removed LNI 192.168.206.117@tcp [12198.119713] Key type .llcrypt unregistered [12198.122787] Key type ._llcrypt unregistered [12287.719509] Key type ._llcrypt registered [12287.728416] Key type .llcrypt registered [12287.915000] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [12288.283288] Key type .llcrypt unregistered [12288.286154] Key type ._llcrypt unregistered [12360.493482] Key type ._llcrypt registered [12360.496292] Key type .llcrypt registered [12360.585300] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [12360.806789] Key type .llcrypt unregistered [12360.814851] Key type ._llcrypt unregistered [12384.517623] Key type ._llcrypt registered [12384.520113] Key type .llcrypt registered [12384.632980] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [12384.913919] Key type .llcrypt unregistered [12384.917235] Key type ._llcrypt unregistered [12410.101297] Key type ._llcrypt registered [12410.107730] Key type .llcrypt registered [12410.224809] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [12410.461240] Key type .llcrypt unregistered [12410.465405] Key type ._llcrypt unregistered [12484.943966] Key type ._llcrypt registered [12484.948694] Key type .llcrypt registered [12485.027524] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [12485.264574] Key type .llcrypt unregistered [12485.269724] Key type ._llcrypt unregistered [12519.474701] Key type ._llcrypt registered [12519.478281] Key type .llcrypt registered [12519.587496] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [12519.942290] Key type .llcrypt unregistered [12519.948603] Key type ._llcrypt unregistered [12548.380235] Key type ._llcrypt registered [12548.383303] Key type .llcrypt registered [12548.492853] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [12548.741753] Key type .llcrypt unregistered [12548.745438] Key type ._llcrypt unregistered [12576.082743] Key type ._llcrypt registered [12576.088512] Key type .llcrypt registered [12576.210764] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [12576.556338] Key type .llcrypt unregistered [12576.564714] Key type ._llcrypt unregistered [12607.686798] Key type ._llcrypt registered [12607.692684] Key type .llcrypt registered [12607.801167] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [12608.127564] Key type .llcrypt unregistered [12608.129289] Key type ._llcrypt unregistered [12632.241390] Key type ._llcrypt registered [12632.244972] Key type .llcrypt registered [12632.320171] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [12632.559610] Key type .llcrypt unregistered [12632.561639] Key type ._llcrypt unregistered [12689.823737] Key type ._llcrypt registered [12689.825734] Key type .llcrypt registered [12689.916522] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [12690.153422] Key type .llcrypt unregistered [12690.157030] Key type ._llcrypt unregistered [12718.003989] Key type ._llcrypt registered [12718.006695] Key type .llcrypt registered [12718.119397] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [12718.421470] Key type .llcrypt unregistered [12718.424284] Key type ._llcrypt unregistered [12747.109385] Key type ._llcrypt registered [12747.111456] Key type .llcrypt registered [12747.202442] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [12747.441321] Key type .llcrypt unregistered [12747.445104] Key type ._llcrypt unregistered [12799.572758] Key type ._llcrypt registered [12799.576441] Key type .llcrypt registered [12799.696918] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [12812.523829] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [12813.544158] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12813.610869] alg: No test for adler32 (adler32-zlib) [12814.630674] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [12814.832947] LNet: Added LNI 192.168.206.117@tcp [8/256/0/180] [12816.495184] Key type lgssc registered [12817.459366] Lustre: Echo OBD driver; http://www.lustre.org/ [12850.883602] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12850.923403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12852.153255] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [12852.208832] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [12852.368176] Lustre: tst250fs-MDT0000: new disk, initializing [12852.430547] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12852.451583] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12856.910524] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [12869.081707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12869.323194] Lustre: 373667:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12869.499460] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [12869.503887] Lustre: Skipped 1 previous similar message [12869.657433] Lustre: tst250fs-MDT0001: new disk, initializing [12869.797983] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12869.851486] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12869.866990] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12875.766708] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [12882.365282] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12890.789420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12890.987455] Lustre: tst250fs-OST0000: new disk, initializing [12890.990669] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [12890.995546] Lustre: 375298:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12891.087912] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12892.358158] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12892.373429] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12892.525553] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12897.133950] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [12907.396988] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12907.596717] Lustre: tst250fs-OST0001: new disk, initializing [12907.599586] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [12907.610646] Lustre: 376168:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12907.699204] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12913.175921] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_default_debug -1 all [12916.814326] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12916.819747] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12916.922839] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12918.107555] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [12920.651372] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [12920.659738] Lustre: Skipped 2 previous similar messages [12937.701345] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12937.704579] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12937.716023] Lustre: Skipped 3 previous similar messages [12937.724477] Lustre: Skipped 2 previous similar messages [12942.817923] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12942.823095] Lustre: Skipped 3 previous similar messages [12942.970034] Lustre: server umount tst250fs-MDT0000 complete [12946.061963] LustreError: 373658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787407242 with bad export cookie 8661005364635965756 [12946.074753] LustreError: MGC192.168.206.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12946.083209] LustreError: 373658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12946.487830] Lustre: server umount tst250fs-MDT0001 complete [12960.364829] Lustre: server umount tst250fs-OST0000 complete [12963.298850] Lustre: 371892:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787407244/real 1787407244] req@ffff9e66b5f9d500 x1874232198888960/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787407260 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12963.331602] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12963.902195] Lustre: server umount tst250fs-OST0001 complete [12983.446467] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [12993.708192] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [13021.790434] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 10:01:56 (1787407316) [13023.590502] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13038.776074] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing set_hostid [13047.766876] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing load_modules_local [13077.865280] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12795 sec ========= 10:02:52 (1787407372) [13079.845715] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 10:02:54 (1787407374) === [13082.991285] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 10:02:57 (1787407377) === [13117.654533] Lustre: DEBUG MARKER: oleg617-server.virtnet: executing unload_modules_local [13120.040980] Key type lgssc unregistered [13120.319376] LNet: 385319:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13120.335307] LNetError: 385319:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13120.347509] LNet: Removed LNI 192.168.206.117@tcp [13121.031161] Key type .llcrypt unregistered [13121.034163] Key type ._llcrypt unregistered