[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 500262696 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002318] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004011] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007026] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008019] pid_max: default: 32768 minimum: 301 [ 0.009163] LSM: Security Framework initializing [ 0.010065] Yama: becoming mindful. [ 0.011043] SELinux: Initializing. [ 0.012101] *** VALIDATE selinux *** [ 0.020642] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025404] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026175] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027121] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028139] *** VALIDATE tmpfs *** [ 0.029431] *** VALIDATE proc *** [ 0.030282] *** VALIDATE cgroup *** [ 0.031010] *** VALIDATE cgroup2 *** [ 0.032235] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033187] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035032] Spectre V2 : User space: Vulnerable [ 0.036010] Speculative Store Bypass: Vulnerable [ 0.039151] debug: unmapping init [mem 0xffffffffa1e59000-0xffffffffa1e60fff] [ 0.042000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042756] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043033] ... version: 2 [ 0.044013] ... bit width: 48 [ 0.045015] ... generic registers: 4 [ 0.046013] ... value mask: 0000ffffffffffff [ 0.047022] ... max period: 00007fffffffffff [ 0.048013] ... fixed-purpose events: 3 [ 0.049012] ... event mask: 000000070000000f [ 0.050295] rcu: Hierarchical SRCU implementation. [ 0.052590] smp: Bringing up secondary CPUs ... [ 0.053575] x86: Booting SMP configuration: [ 0.054025] .... node #0, CPUs: #1 #2 #3 [ 0.057680] smp: Brought up 1 node, 4 CPUs [ 0.059017] smpboot: Max logical packages: 1 [ 0.060019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.139880] node 0 deferred pages initialised in 75ms [ 0.141099] devtmpfs: initialized [ 0.143080] x86/mm: Memory block size: 128MB [ 0.145944] gcov: version magic: 0x41383552 [ 0.147305] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.148143] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.149268] pinctrl core: initialized pinctrl subsystem [ 0.150254] [ 0.150790] ************************************************************* [ 0.151013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.152014] ** ** [ 0.153020] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.154013] ** ** [ 0.155014] ** This means that this kernel is built to expose internal ** [ 0.156015] ** IOMMU data structures, which may compromise security on ** [ 0.157013] ** your system. ** [ 0.158021] ** ** [ 0.159014] ** If you see this message and you are not debugging the ** [ 0.160057] ** kernel, report this immediately to your vendor! ** [ 0.161011] ** ** [ 0.162012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.163018] ************************************************************* [ 0.164750] NET: Registered protocol family 16 [ 0.165534] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.166083] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.167060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.168678] cpuidle: using governor menu [ 0.169626] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.170484] PCI: Using configuration type 1 for base access [ 0.171121] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.179057] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.181021] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.184285] cryptd: max_cpu_qlen set to 1000 [ 0.188158] ACPI: Added _OSI(Module Device) [ 0.189012] ACPI: Added _OSI(Processor Device) [ 0.190011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.192014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.197256] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.202360] ACPI: Interpreter enabled [ 0.203042] ACPI: PM: (supports S0 S3 S4 S5) [ 0.203868] ACPI: Using IOAPIC for interrupt routing [ 0.204141] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.205360] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.212735] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.213033] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.214025] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.215069] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.216966] acpiphp: Slot [2] registered [ 0.217156] acpiphp: Slot [5] registered [ 0.218134] acpiphp: Slot [6] registered [ 0.219069] acpiphp: Slot [7] registered [ 0.219880] acpiphp: Slot [8] registered [ 0.220075] acpiphp: Slot [9] registered [ 0.220909] acpiphp: Slot [10] registered [ 0.221086] acpiphp: Slot [3] registered [ 0.222019] acpiphp: Slot [4] registered [ 0.222939] acpiphp: Slot [11] registered [ 0.223061] acpiphp: Slot [12] registered [ 0.223887] acpiphp: Slot [13] registered [ 0.224089] acpiphp: Slot [14] registered [ 0.225085] acpiphp: Slot [15] registered [ 0.226089] acpiphp: Slot [16] registered [ 0.228120] acpiphp: Slot [17] registered [ 0.229087] acpiphp: Slot [18] registered [ 0.230094] acpiphp: Slot [19] registered [ 0.232126] acpiphp: Slot [20] registered [ 0.233097] acpiphp: Slot [21] registered [ 0.235102] acpiphp: Slot [22] registered [ 0.236104] acpiphp: Slot [23] registered [ 0.238096] acpiphp: Slot [24] registered [ 0.239105] acpiphp: Slot [25] registered [ 0.241123] acpiphp: Slot [26] registered [ 0.243149] acpiphp: Slot [27] registered [ 0.245127] acpiphp: Slot [28] registered [ 0.246098] acpiphp: Slot [29] registered [ 0.248110] acpiphp: Slot [30] registered [ 0.249096] acpiphp: Slot [31] registered [ 0.250068] PCI host bridge to bus 0000:00 [ 0.251016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.253027] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.256060] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.258024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.261026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.263022] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.264167] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.266781] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.268918] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.277014] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.281063] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.284019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.286033] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.287020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.290126] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.291000] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.294045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.297840] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.302013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.312851] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.317016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.322833] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.332017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.338017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.355016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.365074] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.371019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.378036] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.393028] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.403331] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.410020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.416021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.431025] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.440861] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.449013] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.454012] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.467019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.476620] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.482021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.489018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.505023] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.513693] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.518016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.524018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.535017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.548588] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.554674] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.556336] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.558270] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.560144] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.564121] iommu: Default domain type: Passthrough [ 0.565400] SCSI subsystem initialized [ 0.566126] ACPI: bus type USB registered [ 0.567093] usbcore: registered new interface driver usbfs [ 0.568104] usbcore: registered new interface driver hub [ 0.569236] usbcore: registered new device driver usb [ 0.571152] pps_core: LinuxPPS API ver. 1 registered [ 0.573010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.575050] PTP clock support registered [ 0.576136] EDAC MC: Ver: 3.0.0 [ 0.578181] PCI: Using ACPI for IRQ routing [ 0.579893] NetLabel: Initializing [ 0.582011] NetLabel: domain hash size = 128 [ 0.583009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.585106] NetLabel: unlabeled traffic allowed by default [ 0.587298] vgaarb: loaded [ 0.588259] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.590019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.595791] clocksource: Switched to clocksource kvm-clock [ 0.718210] VFS: Disk quotas dquot_6.6.0 [ 0.719863] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.721394] *** VALIDATE ramfs *** [ 0.722167] *** VALIDATE hugetlbfs *** [ 0.723745] pnp: PnP ACPI init [ 0.725353] pnp: PnP ACPI: found 6 devices [ 0.739382] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.741871] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.743280] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.744656] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.746274] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.747952] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.749755] NET: Registered protocol family 2 [ 0.751401] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.754869] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.757992] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.762391] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.764995] TCP: Hash tables configured (established 65536 bind 65536) [ 0.767310] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.769531] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.771650] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.773927] NET: Registered protocol family 1 [ 0.775783] RPC: Registered named UNIX socket transport module. [ 0.777359] RPC: Registered udp transport module. [ 0.778787] RPC: Registered tcp transport module. [ 0.779866] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.781665] NET: Registered protocol family 44 [ 0.783071] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.784889] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.786880] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.788754] PCI: CLS 0 bytes, default 64 [ 0.791133] Unpacking initramfs... [ 2.217767] debug: unmapping init [mem 0xffff9369fcc54000-0xffff9369fffbffff] [ 2.221749] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.223477] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.225822] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.721028] Initialise system trusted keyrings [ 2.722145] Key type blacklist registered [ 2.723891] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.732996] zbud: loaded [ 2.735958] *** VALIDATE nfs *** [ 2.737205] *** VALIDATE nfs4 *** [ 2.738729] pstore: using deflate compression [ 2.741174] Platform Keyring initialized [ 2.833235] NET: Registered protocol family 38 [ 2.834701] Key type asymmetric registered [ 2.835659] Asymmetric key parser 'x509' registered [ 2.836761] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.839705] io scheduler mq-deadline registered [ 2.841207] io scheduler kyber registered [ 2.842856] io scheduler bfq registered [ 2.844168] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.846345] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.848511] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.850527] ACPI: Power Button [PWRF] [ 2.855511] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.860472] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.874089] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.882345] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.897930] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.924682] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.951289] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.955778] Non-volatile memory driver v1.3 [ 2.957196] Linux agpgart interface v0.103 [ 2.981652] virtio_blk virtio1: [vda] 146144 512-byte logical blocks (74.8 MB/71.4 MiB) [ 2.983245] vda: detected capacity change from 0 to 74825728 [ 2.994767] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.997301] vdb: detected capacity change from 0 to 1073741824 [ 3.010110] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.012088] vdc: detected capacity change from 0 to 2621440000 [ 3.026383] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.029154] vdd: detected capacity change from 0 to 2621440000 [ 3.043230] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.046106] vde: detected capacity change from 0 to 4294967296 [ 3.059043] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.061772] vdf: detected capacity change from 0 to 4294967296 [ 3.068846] libphy: Fixed MDIO Bus: probed [ 3.081539] usbcore: registered new interface driver usbserial_generic [ 3.084617] usbserial: USB Serial support registered for generic [ 3.086523] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.091521] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.093536] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.096067] mousedev: PS/2 mouse device common for all mice [ 3.098774] rtc_cmos 00:05: RTC can wake from S4 [ 3.102702] rtc_cmos 00:05: registered as rtc0 [ 3.102835] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.104339] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.110546] intel_pstate: CPU model not supported [ 3.113714] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.117575] hid: raw HID events driver (C) Jiri Kosina [ 3.119735] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.121429] usbcore: registered new interface driver usbhid [ 3.121438] usbhid: USB HID core driver [ 3.121587] drop_monitor: Initializing network drop monitor service [ 3.121742] Initializing XFRM netlink socket [ 3.122146] NET: Registered protocol family 10 [ 3.127932] Segment Routing with IPv6 [ 3.134972] NET: Registered protocol family 17 [ 3.136717] mpls_gso: MPLS GSO support [ 3.142098] RAS: Correctable Errors collector initialized. [ 3.143666] AVX version of gcm_enc/dec engaged. [ 3.144783] AES CTR mode by8 optimization enabled [ 3.233670] sched_clock: Marking stable (3233639221, 0)->(4178093321, -944454100) [ 3.236753] registered taskstats version 1 [ 3.239603] Loading compiled-in X.509 certificates [ 3.242471] zswap: loaded using pool lzo/zbud [ 3.273682] Key type big_key registered [ 3.288080] Key type encrypted registered [ 3.289644] ima: No TPM chip found, activating TPM-bypass! [ 3.291266] ima: Allocated hash algorithm: sha1 [ 3.292491] ima: No architecture policies found [ 3.294121] evm: Initialising EVM extended attributes: [ 3.295566] evm: security.selinux [ 3.296493] evm: security.ima [ 3.297889] evm: security.capability [ 3.298939] evm: HMAC attrs: 0x1 [ 3.301345] rtc_cmos 00:05: setting system clock to 2026-08-23 07:17:47 UTC (1787469467) [ 3.307670] debug: unmapping init [mem 0xffffffffa2e03000-0xffffffffa2ffffff] [ 3.310633] debug: unmapping init [mem 0xffffffffa1b82000-0xffffffffa1e58fff] [ 3.320070] Write protecting the kernel read-only data: 28672k [ 3.322447] debug: unmapping init [mem 0xffffffffa0203000-0xffffffffa03fffff] [ 3.324462] debug: unmapping init [mem 0xffffffffa0b14000-0xffffffffa0bfffff] [ 3.354332] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.363734] systemd[1]: Detected virtualization kvm. [ 3.365538] systemd[1]: Detected architecture x86-64. [ 3.367365] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.390525] systemd[1]: No hostname configured. [ 3.392094] systemd[1]: Set hostname to . [ 3.393923] random: systemd: uninitialized urandom read (16 bytes read) [ 3.395667] systemd[1]: Initializing machine ID from random generator. [ 3.553550] random: systemd: uninitialized urandom read (16 bytes read) [ 3.556553] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.561365] random: systemd: uninitialized urandom read (16 bytes read) [ 3.563731] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.568495] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local File Systems. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Sockets. Starting Setup Virtual Console... [ OK ] Started Create Volatile Files and Directories. [ OK [ 3.694896] hrtimer: interrupt took 3805270 ns ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.281291] device-mapper: uevent: version 1.0.3 [ 4.282761] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.961094] random: fast init done Starting dracut initqueue hook... [ 5.007975] virtio_net virtio0 ens2: renamed from eth0 [ 5.135959] scsi host0: ata_piix [ 5.198888] scsi host1: ata_piix [ 5.200469] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.202414] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.459486] dracut-initqueue[594]: RTNETLINK answers: File exists [ 9.806136] random: crng init done [ 9.807578] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.304894] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.480309] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.736701] SELinux: Disabled at runtime. [ 11.797232] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.805343] systemd[1]: Detected virtualization kvm. [ 11.807179] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.323221] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.327115] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.334120] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.338538] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.342105] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.349751] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.361269] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-getty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ 12.430570] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... Mounting Huge Pages File System... Starting Apply Kernel Variables... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on Process Core Dump Socket. Starting Remount Root and Kernel File Systems... [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 12.804461] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ OK ] Started udev Coldplug all Devices. [ 13.106957] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.120703] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.286688] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.296905] EDAC sbridge: Ver: 1.1.2 [ 14.963824] Key type dns_resolver registered [ 15.260209] NFS: Registering the id_resolver key type [ 15.262226] Key type id_resolver registered [ 15.263839] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg224-server login: [ 42.063946] libcfs: loading out-of-tree module taints kernel. [ 42.084128] Key type ._llcrypt registered [ 42.085678] Key type .llcrypt registered [ 42.140755] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [ 62.494204] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 64.550722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 64.569673] alg: No test for adler32 (adler32-zlib) [ 66.610622] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 67.951698] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 69.919241] Key type lgssc registered [ 72.145318] Lustre: Echo OBD driver; http://www.lustre.org/ [ 93.606428] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 141.532066] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 162.459634] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 162.538400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 164.026444] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 164.066665] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 164.204184] Lustre: lustre-MDT0000: new disk, initializing [ 164.391488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 164.442871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 169.686974] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 185.550729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 185.725546] Lustre: 6516:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 185.799404] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 185.808705] Lustre: Skipped 1 previous similar message [ 185.900714] Lustre: lustre-MDT0001: new disk, initializing [ 185.964887] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 185.986214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 186.006892] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 190.286800] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 195.007730] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 206.234507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 206.535922] Lustre: lustre-OST0000: new disk, initializing [ 206.539296] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 206.546933] Lustre: 8455:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 206.639809] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 213.071816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 213.084182] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 213.180422] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 213.197528] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 229.340790] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 229.473084] Lustre: lustre-OST0001: new disk, initializing [ 229.477813] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 229.488577] Lustre: 9528:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 229.547217] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 236.110406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 236.126111] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 236.243707] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 236.447136] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 250.659381] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 261.609362] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 268.710186] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing check_logdir /tmp/testlogs/ [ 274.971862] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing yml_node [ 281.246850] Lustre: DEBUG MARKER: Client: 2.17.57.63 [ 284.504512] Lustre: DEBUG MARKER: MDS: 2.17.57.63 [ 287.304472] Lustre: DEBUG MARKER: OSS: 2.17.57.63 [ 289.485360] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Aug 23 03:22:32 EDT 2026 [ 309.823442] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 311.815627] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 328.675115] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 328.683044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 328.692777] Lustre: Skipped 1 previous similar message [ 333.795153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 333.803109] Lustre: Skipped 5 previous similar messages [ 335.361644] Lustre: server umount lustre-MDT0000 complete [ 339.175548] LustreError: 6508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787469803 with bad export cookie 6341229166043642893 [ 339.176829] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 339.191312] LustreError: 6508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 344.032245] LustreError: 8462:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 344.037809] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 344.049820] LustreError: 8462:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 344.062755] Lustre: Skipped 2 previous similar messages [ 344.075728] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 344.075728] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 345.724744] Lustre: server umount lustre-MDT0001 complete [ 349.748734] Lustre: server umount lustre-OST0000 complete [ 353.985449] Lustre: server umount lustre-OST0001 complete [ 361.927473] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [ 373.545650] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 418.406800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 418.796954] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 418.860515] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 418.963862] Lustre: lustre-MDT0000: new disk, initializing [ 419.040394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 419.067918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 424.247599] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 437.143517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 437.340423] Lustre: 15083:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 437.409384] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 437.420611] Lustre: Skipped 1 previous similar message [ 437.566336] Lustre: lustre-MDT0001: new disk, initializing [ 437.734227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 437.803477] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 437.819495] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 443.537396] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 457.112741] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 464.649513] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 472.397634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 472.816405] Lustre: lustre-OST0000: new disk, initializing [ 472.830926] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 472.841265] Lustre: 17052:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 472.989717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 473.939937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 473.951885] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 474.123864] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 479.871351] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 492.000409] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 497.246864] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 497.429220] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 501.930417] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 502.115702] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 504.804574] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 504.815140] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 504.828023] Lustre: Skipped 1 previous similar message [ 504.838132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 504.864251] Lustre: Skipped 1 previous similar message [ 509.929682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 509.938911] Lustre: Skipped 1 previous similar message [ 510.237100] Lustre: server umount lustre-OST0000 complete [ 515.043744] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 515.055133] Lustre: Skipped 2 previous similar messages [ 520.163142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 520.170380] Lustre: Skipped 3 previous similar messages [ 529.375311] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 529.592228] Lustre: server umount lustre-MDT0000 complete [ 530.402059] LustreError: 15096:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 530.448383] LustreError: 15096:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 533.868549] LustreError: 15076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787469998 with bad export cookie 6341229166043644426 [ 533.885658] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 533.897575] LustreError: 15076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 534.460376] Lustre: server umount lustre-MDT0001 complete [ 541.307330] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 03:26:44 (1787470004) [ 561.978048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 562.419185] LustreError: 19928:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 562.493071] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 567.620573] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 567.786672] LustreError: 19929:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 571.874961] LustreError: 19928:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 578.311954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 583.635783] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 591.691671] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 598.304732] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 600.932750] Lustre: Failing over lustre-MDT0000 [ 601.323516] Lustre: server umount lustre-MDT0000 complete [ 603.063135] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 603.072615] LustreError: 19925:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 603.082975] Lustre: Skipped 1 previous similar message [ 603.117272] LustreError: 19925:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 619.424182] Lustre: 3648:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787470067/real 1787470067] req@ffff936947990700 x1874297652156288/t0(0) o400->MGC192.168.202.124@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787470083 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 619.489546] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 619.507380] LustreError: 19925:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 619.526420] LustreError: 19925:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 623.090549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 629.925517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 629.929305] Lustre: Skipped 1 previous similar message [ 629.973476] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 635.078635] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 635.373270] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 635.374420] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 635.436726] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 645.745025] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 650.721171] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 650.729024] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 650.748664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 650.753351] Lustre: Skipped 2 previous similar messages [ 656.332618] Lustre: server umount lustre-MDT0000 complete [ 658.916809] LustreError: 19929:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 658.935837] LustreError: 19929:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 660.128608] LustreError: 19909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787470124 with bad export cookie 6341229166043645840 [ 660.142392] LustreError: 19909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 660.162605] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 660.509188] Lustre: server umount lustre-MDT0001 complete [ 669.556198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 675.234527] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 684.697620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 689.692302] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 697.605445] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 703.600332] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 705.857128] Lustre: Failing over lustre-MDT0000 [ 706.022492] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 706.041965] Lustre: Skipped 1 previous similar message [ 706.315551] Lustre: server umount lustre-MDT0000 complete [ 722.144700] Lustre: 3648:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787470170/real 1787470170] req@ffff936945a71c00 x1874297652185088/t0(0) o400->MGC192.168.202.124@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787470186 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 722.191745] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 725.456962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 727.521706] LustreError: 23263:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 727.535974] LustreError: 23263:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 733.063286] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 738.025392] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 738.272850] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 738.289421] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 738.295619] Lustre: Skipped 1 previous similar message [ 738.308725] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 747.717521] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 753.635178] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 753.643375] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 753.655814] Lustre: Skipped 1 previous similar message [ 753.660350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 753.685514] Lustre: Skipped 3 previous similar messages [ 758.397804] Lustre: server umount lustre-MDT0000 complete [ 761.962651] LustreError: 23248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787470226 with bad export cookie 6341229166043646778 [ 761.971498] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 761.974224] LustreError: 23248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 762.608241] Lustre: server umount lustre-MDT0001 complete [ 772.516148] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 03:30:35 (1787470235) [ 791.531747] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [ 802.607395] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 849.474640] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 861.121173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 861.433276] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 861.514923] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 861.586719] Lustre: lustre-MDT0000: new disk, initializing [ 861.684923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 861.694581] Lustre: Skipped 3 previous similar messages [ 861.713503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 866.543493] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 878.742203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 878.885772] Lustre: 29641:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 879.083188] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 884.573646] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 895.788340] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 903.032684] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 910.576960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 910.865688] Lustre: 31609:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 912.517531] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 912.599697] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 912.688569] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 912.707651] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 918.380639] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 929.026474] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 935.138880] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 958.945474] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 958.963195] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 958.964096] LustreError: Skipped 1 previous similar message [ 958.965651] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 958.965657] Lustre: Skipped 2 previous similar messages [ 958.986506] Lustre: Skipped 2 previous similar messages [ 965.133363] Lustre: server umount lustre-OST0000 complete [ 975.406303] Lustre: server umount lustre-MDT0000 complete [ 979.425537] LustreError: 29651:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 979.445530] LustreError: 29651:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 979.588537] LustreError: 29634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787470443 with bad export cookie 6341229166043647177 [ 979.597156] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 979.606090] LustreError: 29634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 980.018784] Lustre: server umount lustre-MDT0001 complete [ 994.801585] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 998.362302] Key type lgssc unregistered [ 998.750799] LNet: 33598:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 998.764919] LNetError: 33598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 998.774993] LNet: Removed LNI 192.168.202.124@tcp [ 1000.043733] Key type .llcrypt unregistered [ 1000.046082] Key type ._llcrypt unregistered [ 1026.417039] Key type ._llcrypt registered [ 1026.421234] Key type .llcrypt registered [ 1026.552907] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 1027.832624] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1027.858440] alg: No test for adler32 (adler32-zlib) [ 1028.870967] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 1029.080141] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 1030.711215] Key type lgssc registered [ 1031.974731] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1040.448467] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1040.481083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1041.983833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1047.219100] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 1052.140739] LustreError: 35013:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1052.164708] LustreError: 35013:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1056.688989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1057.135509] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1062.496229] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 1070.654164] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1076.663639] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1084.871131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1085.197216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1087.297122] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1091.616236] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 1099.950332] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1102.642411] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1119.353773] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 03:36:22 (1787470582) [ 1121.676181] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1123.796483] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1131.273757] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 03:36:34 (1787470594) [ 1141.517296] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 03:36:44 (1787470604) [ 1142.913366] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1152.067341] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1152.077773] Lustre: Skipped 2 previous similar messages [ 1154.324697] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 03:36:57 (1787470617) [ 1156.809955] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1156.817944] Lustre: Skipped 2 previous similar messages [ 1165.858771] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1165.865121] Lustre: Skipped 4 previous similar messages [ 1183.458270] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 03:37:26 (1787470646) [ 1186.939899] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1190.928247] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1200.252964] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 03:37:43 (1787470663) [ 1202.432847] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1202.440225] Lustre: Skipped 6 previous similar messages [ 1217.832869] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 03:38:00 (1787470680) [ 1235.937773] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1235.958084] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1235.978580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1235.990335] Lustre: Skipped 1 previous similar message [ 1241.060204] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1241.065380] Lustre: Skipped 1 previous similar message [ 1246.185586] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1246.200494] Lustre: Skipped 1 previous similar message [ 1250.271358] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1250.924367] Lustre: server umount lustre-OST0000 complete [ 1256.928769] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1256.938206] LustreError: Skipped 1 previous similar message [ 1256.942370] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1256.954542] Lustre: Skipped 1 previous similar message [ 1256.959255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1262.054020] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1262.055709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1262.098890] Lustre: Skipped 1 previous similar message [ 1263.458040] Lustre: server umount lustre-MDT0000 complete [ 1267.191067] LustreError: 38339:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1267.223656] LustreError: 38339:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1268.932354] LustreError: 40045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787470733 with bad export cookie 1850420816352750042 [ 1268.937329] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1268.947656] LustreError: 40045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1269.437525] Lustre: server umount lustre-MDT0001 complete [ 1285.963435] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 1289.499493] Key type lgssc unregistered [ 1289.864987] LNet: 42880:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1289.874725] LNetError: 42880:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1289.892691] LNet: Removed LNI 192.168.202.124@tcp [ 1291.060948] Key type .llcrypt unregistered [ 1291.063963] Key type ._llcrypt unregistered [ 1312.239918] Key type ._llcrypt registered [ 1312.242490] Key type .llcrypt registered [ 1312.337326] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [ 1331.812676] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 1333.190467] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1333.226476] alg: No test for adler32 (adler32-zlib) [ 1334.351425] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 1334.607681] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 1336.289395] Key type lgssc registered [ 1337.500602] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1377.796805] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1377.841665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1379.216931] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1379.243993] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1379.328619] Lustre: lustre-MDT0000: new disk, initializing [ 1379.417356] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1379.440462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1384.747416] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 1396.552394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1396.783375] Lustre: 46039:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1396.878708] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1396.883297] Lustre: Skipped 1 previous similar message [ 1396.957538] Lustre: lustre-MDT0001: new disk, initializing [ 1397.127310] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1397.212456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1397.240556] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1402.504307] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 1413.413054] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1421.909438] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1429.583965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1429.835537] Lustre: lustre-OST0000: new disk, initializing [ 1429.840422] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1429.848175] Lustre: 48005:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1429.965599] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1436.233561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1436.241038] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1436.408713] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1436.423722] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 1446.977958] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1453.064526] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1453.283517] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1458.049492] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1458.294165] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1462.241820] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1462.245897] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1462.275992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1462.303843] Lustre: Skipped 1 previous similar message [ 1466.678540] Lustre: server umount lustre-OST0000 complete [ 1472.480858] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1472.504627] Lustre: Skipped 1 previous similar message [ 1472.508625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1472.516801] Lustre: Skipped 1 previous similar message [ 1477.600717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1477.604775] Lustre: Skipped 1 previous similar message [ 1482.721808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1482.729701] Lustre: Skipped 1 previous similar message [ 1485.279167] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1485.869555] Lustre: server umount lustre-MDT0000 complete [ 1490.657287] LustreError: 46030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787470954 with bad export cookie 12371248988126419965 [ 1490.661524] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1490.683264] LustreError: 46030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1491.366715] Lustre: server umount lustre-MDT0001 complete [ 1503.971481] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 03:42:47 (1787470967) [ 1511.798666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1512.266125] LustreError: 50466:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1512.282843] LustreError: 50466:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1512.369685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1517.375349] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 1517.536382] LustreError: 50467:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1522.663970] LustreError: 50466:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1527.478928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1527.790051] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1527.954907] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1532.649857] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 1543.122710] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1549.570395] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1559.134593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1559.525389] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1568.280716] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 1577.166859] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1586.811803] LustreError: 53000:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1588.307666] Lustre: Modifying parameter general.timeout=1 in log params [ 1590.048879] Lustre: Modifying parameter general.timeout=2 in log params [ 1592.203275] Lustre: Modifying parameter general.timeout=4 in log params [ 1592.209571] Lustre: Skipped 1 previous similar message [ 1596.815605] Lustre: Modifying parameter general.timeout=8 in log params [ 1596.821699] Lustre: Skipped 3 previous similar messages [ 1605.603358] Lustre: Modifying parameter general.timeout=15 in log params [ 1605.606033] Lustre: Skipped 6 previous similar messages [ 1621.793931] Lustre: Modifying parameter general.timeout=30 in log params [ 1621.800312] Lustre: Skipped 14 previous similar messages [ 1654.535911] Lustre: Modifying parameter general.timeout=60 in log params [ 1654.540538] Lustre: Skipped 29 previous similar messages [ 1718.732173] Lustre: Modifying parameter general.timeout=121 in log params [ 1718.735361] Lustre: Skipped 60 previous similar messages [ 1740.693420] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 03:46:43 (1787471203) [ 1744.090790] Lustre: 59958:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1746.535610] Lustre: 60059:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1748.707510] Lustre: 60169:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1754.536214] Lustre: 60479:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1754.549186] Lustre: 60479:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1785.809617] Lustre: 61840:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1785.831530] LustreError: 61840:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1791.151986] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 03:47:33 (1787471253) [ 1808.867878] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1808.887196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1808.892893] Lustre: Skipped 3 previous similar messages [ 1808.906460] Lustre: Skipped 2 previous similar messages [ 1811.991540] Lustre: server umount lustre-MDT0000 complete [ 1813.984605] LustreError: 55704:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1814.014719] LustreError: 55704:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1815.708299] LustreError: 50447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787471279 with bad export cookie 12371248988126420847 [ 1815.712646] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1815.738054] LustreError: 50447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1816.168827] Lustre: server umount lustre-MDT0001 complete [ 1830.133139] Lustre: server umount lustre-OST0000 complete [ 1860.882642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1861.205416] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1861.250042] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1861.259839] Lustre: Skipped 31 previous similar messages [ 1861.593056] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1866.775619] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 1876.933431] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1877.072726] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1877.104720] Lustre: 64373:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1881.576818] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 1889.958332] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1896.930807] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1904.600959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1904.818807] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1905.207483] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1905.213201] Lustre: Skipped 1 previous similar message [ 1912.127398] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 1920.298271] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1922.451370] Lustre: 66940:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1932.265149] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1932.279978] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1932.290816] Lustre: Skipped 1 previous similar message [ 1932.292847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1932.300403] Lustre: Skipped 1 previous similar message [ 1937.054421] Lustre: server umount lustre-OST0000 complete [ 1942.499777] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1942.520192] Lustre: Skipped 1 previous similar message [ 1947.620024] LustreError: 64380:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1947.645169] Lustre: server umount lustre-MDT0000 complete [ 1947.646339] LustreError: 64380:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1952.573547] LustreError: 64365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787471416 with bad export cookie 12371248988126424060 [ 1952.575191] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1952.585265] LustreError: 64365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1952.946674] Lustre: server umount lustre-MDT0001 complete [ 1965.001351] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 1967.149876] Key type lgssc unregistered [ 1967.415826] LNet: 68155:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1967.425705] LNetError: 68155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1967.435877] LNet: Removed LNI 192.168.202.124@tcp [ 1968.216092] Key type .llcrypt unregistered [ 1968.218216] Key type ._llcrypt unregistered [ 1995.360862] Key type ._llcrypt registered [ 1995.365687] Key type .llcrypt registered [ 1995.539650] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 1996.739912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1996.773781] alg: No test for adler32 (adler32-zlib) [ 1997.863577] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 1998.050338] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 1999.687496] Key type lgssc registered [ 2000.765182] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2008.767292] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2008.802526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2010.441112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2014.476899] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2019.812775] LustreError: 69571:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2019.850711] LustreError: 69571:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2024.595977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2025.192539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2030.446599] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2038.968181] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2046.150791] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2054.949919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2055.145447] LustreError: 71398:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2055.669718] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2062.317615] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2070.710836] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2079.819988] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2090.475122] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2090.481455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2090.492476] Lustre: Skipped 1 previous similar message [ 2090.517204] Lustre: Skipped 2 previous similar messages [ 2095.588321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2095.595300] Lustre: Skipped 2 previous similar messages [ 2095.729946] Lustre: server umount lustre-MDT0000 complete [ 2099.295102] LustreError: 72256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787471563 with bad export cookie 761089071839698046 [ 2099.298139] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2099.306742] LustreError: 72256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2099.543917] Lustre: server umount lustre-MDT0001 complete [ 2114.292899] Lustre: server umount lustre-OST0000 complete [ 2144.502170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2144.904351] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2144.932299] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2145.327071] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2150.055460] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2161.234860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2161.322845] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2161.348714] Lustre: 74444:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2162.007603] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2167.883533] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2176.837230] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2184.060939] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2193.123794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2193.406715] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2193.774278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2200.566688] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2209.306621] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2211.448495] Lustre: 77021:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2213.413535] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2223.590982] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2223.600068] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2223.609959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2223.619633] Lustre: Skipped 2 previous similar messages [ 2223.637407] Lustre: Skipped 1 previous similar message [ 2225.286581] Lustre: server umount lustre-OST0000 complete [ 2233.312351] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2233.319615] LustreError: Skipped 1 previous similar message [ 2233.323568] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2233.337370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2236.086055] Lustre: server umount lustre-MDT0000 complete [ 2238.950904] LustreError: 74451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2238.991740] LustreError: 74451:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2240.009879] LustreError: 77024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787471704 with bad export cookie 761089071839698893 [ 2240.021057] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2240.028952] LustreError: 77024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2240.507177] Lustre: server umount lustre-MDT0001 complete [ 2255.257888] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 2258.452300] Key type lgssc unregistered [ 2258.807030] LNet: 78238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2258.818649] LNetError: 78238:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2258.848743] LNet: Removed LNI 192.168.202.124@tcp [ 2260.002407] Key type .llcrypt unregistered [ 2260.007922] Key type ._llcrypt unregistered [ 2292.861317] Key type ._llcrypt registered [ 2292.867571] Key type .llcrypt registered [ 2293.033884] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 2294.598984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2294.678332] alg: No test for adler32 (adler32-zlib) [ 2295.850657] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 2296.155687] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 2297.863187] Key type lgssc registered [ 2299.602989] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2308.936778] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2308.971111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2310.682391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2317.895767] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2319.851894] LustreError: 79652:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2319.888633] LustreError: 79652:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2324.965609] LustreError: 79653:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2327.663788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2329.156685] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2333.464147] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2342.755077] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2349.309503] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2356.801846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2357.425410] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2364.086421] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2372.038816] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2388.739827] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2389.591882] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2391.076232] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2391.085263] Lustre: Skipped 1 previous similar message [ 2393.750304] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2393.753815] Lustre: Skipped 2 previous similar messages [ 2398.673966] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2398.677813] Lustre: Skipped 4 previous similar messages [ 2407.016012] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2407.018825] Lustre: Skipped 8 previous similar messages [ 2423.070274] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2423.076593] Lustre: Skipped 16 previous similar messages [ 2455.517288] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2455.520667] Lustre: Skipped 37 previous similar messages [ 2485.512182] Lustre: 87775:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2488.359599] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 03:59:11 (1787471951) [ 2489.698543] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2491.533890] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 03:59:15 (1787471955) [ 2497.000218] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2497.006305] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2497.021778] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2498.019350] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2498.020325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2498.029209] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2498.047522] Lustre: Skipped 1 previous similar message [ 2501.657333] Lustre: server umount lustre-OST0000 complete [ 2507.749210] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2507.760403] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2507.777343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2511.620188] Lustre: server umount lustre-MDT0000 complete [ 2513.378076] LustreError: 79649:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2513.400860] LustreError: 79649:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2515.417230] LustreError: 83450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787471979 with bad export cookie 2786863235224287230 [ 2515.439749] LustreError: 83450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2515.440863] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2515.636262] Lustre: server umount lustre-MDT0001 complete [ 2527.489536] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 2529.999681] Key type lgssc unregistered [ 2530.358531] LNet: 89028:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2530.365901] LNetError: 89028:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2530.393386] LNet: Removed LNI 192.168.202.124@tcp [ 2531.338271] Key type .llcrypt unregistered [ 2531.341710] Key type ._llcrypt unregistered [ 2537.431765] Key type ._llcrypt registered [ 2537.434424] Key type .llcrypt registered [ 2537.563530] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2543.280739] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules [ 2544.735103] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2544.946242] alg: No test for adler32 (adler32-zlib) [ 2545.963860] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 2546.161572] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 2547.791205] Key type lgssc registered [ 2549.090917] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2550.350852] LustreError: 90232:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2571.003669] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 2573.206194] LustreError: 90232:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2581.201378] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2581.229564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2583.124768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2588.489354] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2593.250268] LustreError: 91258:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2593.285684] LustreError: 91258:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2598.369703] LustreError: 91257:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2599.053693] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 04:01:01 (1787472061) [ 2603.499241] LustreError: 91258:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2604.111179] LustreError: 91289:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2604.124139] LustreError: 91289:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2604.131687] LustreError: 91289:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 2604.754539] Lustre: server umount lustre-MDT0000 complete [ 2621.283480] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 2623.623433] Key type lgssc unregistered [ 2623.966050] LNet: 92798:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2623.975132] LNetError: 92798:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2623.991127] LNet: Removed LNI 192.168.202.124@tcp [ 2624.854142] Key type .llcrypt unregistered [ 2624.858965] Key type ._llcrypt unregistered [ 2647.731396] Key type ._llcrypt registered [ 2647.733417] Key type .llcrypt registered [ 2647.834465] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 2649.053772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2649.154752] alg: No test for adler32 (adler32-zlib) [ 2650.361600] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 2650.569475] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 2652.207139] Key type lgssc registered [ 2653.238794] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2661.026240] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2661.049870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2662.803125] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2668.825473] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2672.109216] LustreError: 94015:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2672.140383] LustreError: 94015:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2677.221439] LustreError: 94016:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2678.426477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2679.263017] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2684.161624] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2692.874414] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2700.092161] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2708.307640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2708.862368] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2715.734859] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2725.147828] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2787.194622] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 04:04:10 (1787472250) [ 2789.305638] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2791.794577] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 04:04:14 (1787472254) [ 2799.076824] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2799.084302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2799.102904] Lustre: Skipped 2 previous similar messages [ 2799.123518] Lustre: Skipped 2 previous similar messages [ 2804.195230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2804.209219] Lustre: Skipped 2 previous similar messages [ 2804.600278] Lustre: server umount lustre-MDT0000 complete [ 2808.958165] LustreError: 93997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787472273 with bad export cookie 5384954547286824419 [ 2808.980021] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2809.282425] Lustre: server umount lustre-MDT0001 complete [ 2824.451464] Lustre: server umount lustre-OST0000 complete [ 2837.048886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2837.464742] LustreError: 98336:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2837.637206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2842.125422] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2843.105600] LustreError: 98337:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2847.202051] LustreError: 98336:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2852.956970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2853.751201] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2858.896285] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2868.958265] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2878.486773] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2894.131583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2894.387349] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2894.393537] LustreError: 98325:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2894.416443] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2894.422428] LustreError: 100420:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2894.428744] LustreError: 100420:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2894.433577] LustreError: 100420:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2894.490551] Lustre: server umount lustre-OST0000 complete [ 2894.494539] LustreError: 100420:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2903.226352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2903.422150] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2903.429174] LustreError: 98324:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2903.448419] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2903.455169] LustreError: 100887:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2903.460552] LustreError: 100887:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2903.466972] LustreError: 100887:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2903.529538] Lustre: server umount lustre-OST0000 complete [ 2903.532234] LustreError: 100887:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2913.699711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2913.872199] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2913.880028] Lustre: Found index 0 for lustre-OST0000, updating log [ 2913.885427] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2913.976403] Lustre: lustre-OST0000: new disk, initializing [ 2913.979970] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2913.987090] Lustre: 101423:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2914.247892] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2921.439916] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 2925.822065] Lustre: Failing over lustre-OST0000 [ 2925.996706] Lustre: server umount lustre-OST0000 complete [ 2933.734517] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2933.742714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2933.752796] Lustre: Skipped 1 previous similar message [ 2938.858038] LustreError: 98336:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2938.898227] LustreError: 98336:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2938.954239] Lustre: server umount lustre-MDT0000 complete [ 2943.755161] LustreError: 98317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787472407 with bad export cookie 5384954547286827163 [ 2943.773158] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2943.783800] LustreError: 98317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 2954.462221] Lustre: server umount lustre-MDT0001 complete [ 2965.219777] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 04:07:08 (1787472428) [ 2988.956945] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 3000.592685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3000.886516] LustreError: 103792:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3001.012422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3006.564967] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3016.283328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3022.530437] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3026.236620] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 3033.866935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3034.487386] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3034.494132] Lustre: Skipped 1 previous similar message [ 3034.513085] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3041.482983] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3041.655266] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3041.701635] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 3041.701790] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3052.626522] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3052.858741] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3052.988433] Lustre: lustre-OST0001: new disk, initializing [ 3052.991418] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3052.998141] Lustre: 106109:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3060.382525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3060.397924] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3060.572541] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3061.873972] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3071.032769] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3079.392569] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3083.749659] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 3083.754412] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3083.781698] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3083.790170] Lustre: Skipped 1 previous similar message [ 3086.330141] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3086.362034] Lustre: Skipped 1 previous similar message [ 3090.056443] Lustre: server umount lustre-MDT0001 complete [ 3090.431710] LustreError: 105658:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3090.455560] LustreError: 105658:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3101.139860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3101.225582] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3101.230179] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3101.268281] Lustre: 103781:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3101.833567] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3101.841653] Lustre: Skipped 1 previous similar message [ 3103.031717] LustreError: 107688:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3103.045503] LustreError: 107688:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3103.053790] LustreError: 103784:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3103.143474] LustreError: 107693:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.124@tcp: cfg command failed: rc = -17 [ 3103.150828] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3103.150828] [ 3106.144706] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3107.319894] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3119.418310] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 04:09:42 (1787472582) [ 3142.683925] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 3163.804422] Lustre: 109747:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3188.667423] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3193.174086] Lustre: 110881:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3280.574569] Lustre: server umount lustre-MDT0000 complete [ 3280.872615] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3280.887098] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3280.921983] LustreError: 103793:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3280.956546] LustreError: 103793:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 3285.407861] LustreError: 103773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787472749 with bad export cookie 5384954547286828164 [ 3285.413436] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3285.420496] LustreError: 103773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3286.326608] Lustre: server umount lustre-MDT0001 complete [ 3302.687293] Lustre: 93326:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787472750/real 1787472750] req@ffff936953aa6680 x1874300359782528/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787472766 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3302.756529] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3302.798565] Lustre: Skipped 3 previous similar messages [ 3303.286816] Lustre: server umount lustre-OST0000 complete [ 3305.887261] Lustre: 93327:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787472754/real 1787472754] req@ffff936953b64380 x1874300359782784/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787472770 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3306.981285] Lustre: 93326:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787472755/real 1787472755] req@ffff936953fd0380 x1874300359783040/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787472771 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3309.337238] Lustre: server umount lustre-OST0001 complete [ 3315.320197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3327.447141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3336.476546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3348.634700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3372.922200] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 3386.002266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3386.036387] Lustre: lustre-MDT0000: reset Object Index mappings [ 3386.530538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3391.959406] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3403.619187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3403.632156] Lustre: lustre-MDT0001: reset Object Index mappings [ 3409.807932] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3413.830380] Lustre: 115986:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3422.007754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3429.250468] LustreError: 116339:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3429.264073] LustreError: 116339:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 3429.273806] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:232 to 0x280000400:257) [ 3431.380582] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:350 to 0x280000401:386) [ 3431.565364] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3442.746664] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3448.837084] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:199 to 0x2c0000400:225) [ 3448.843394] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3451.911815] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3461.641724] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3465.769601] Lustre: 117862:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3521.468184] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 04:16:24 (1787472984) [ 3547.104680] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3547.117427] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3547.131826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3547.134416] Lustre: Skipped 4 previous similar messages [ 3553.664431] Lustre: server umount lustre-MDT0000 complete [ 3557.796848] LustreError: 114817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787473021 with bad export cookie 5384954547286875008 [ 3557.815299] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3561.457371] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3561.467077] Lustre: Skipped 4 previous similar messages [ 3564.467968] Lustre: server umount lustre-MDT0001 complete [ 3575.017158] Lustre: server umount lustre-OST0000 complete [ 3586.061892] Lustre: server umount lustre-OST0001 complete [ 3594.334582] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [ 3606.939586] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 3648.990431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3649.392200] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3649.430647] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3649.517375] Lustre: lustre-MDT0000: new disk, initializing [ 3649.638473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3649.653179] Lustre: Skipped 3 previous similar messages [ 3649.683316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3656.154549] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3666.902993] Lustre: server umount lustre-MDT0000 complete [ 3680.206497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3680.606952] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3680.652280] Lustre: 122139:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3680.679375] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3685.666927] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3690.959828] Lustre: server umount lustre-MDT0000 complete [ 3701.589965] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 04:19:24 (1787473164) [ 3703.370501] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3705.659506] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 04:19:28 (1787473168) [ 3724.922789] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 3735.096483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3740.535800] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3750.275501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3750.462448] Lustre: 124191:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3750.494118] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3750.499275] Lustre: Skipped 1 previous similar message [ 3750.546692] Lustre: lustre-MDT0001: new disk, initializing [ 3750.682968] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3750.699782] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3757.535462] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3764.363510] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3774.039798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3774.524110] Lustre: lustre-OST0000: new disk, initializing [ 3774.532695] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3774.546706] Lustre: 125776:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3775.789344] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3775.794163] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3776.078013] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3785.490469] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3799.846368] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3800.002901] Lustre: lustre-OST0001: new disk, initializing [ 3800.005822] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3800.010634] Lustre: 126646:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3801.362017] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3801.405420] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3801.536408] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3807.731591] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 3818.931797] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3824.031615] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4246.229272] Lustre: 133067:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4246.237609] Lustre: 133067:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4246.251649] Lustre: 133067:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4246.257950] Lustre: 133067:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4246.274780] Lustre: 133067:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4246.289599] Lustre: 133067:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4517.682540] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 04:33:00 (1787473980) [ 4533.751181] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4533.772061] Lustre: Skipped 8 previous similar messages [ 4533.780579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4533.787576] Lustre: Skipped 4 previous similar messages [ 4538.851941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4538.867765] Lustre: Skipped 4 previous similar messages [ 4539.389641] Lustre: server umount lustre-MDT0000 complete [ 4543.094504] LustreError: 124947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787474007 with bad export cookie 5384954547286943916 [ 4543.100217] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4543.105098] LustreError: 124947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4543.968648] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4543.973765] LustreError: 124197:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4543.973863] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4543.987760] Lustre: Skipped 1 previous similar message [ 4544.000788] LustreError: 124197:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 4549.756484] Lustre: server umount lustre-MDT0001 complete [ 4560.000629] Lustre: server umount lustre-OST0000 complete [ 4578.273959] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4578.580678] Lustre: server umount lustre-OST0001 complete [ 4586.205581] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [ 4602.605329] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 4643.828537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4644.060773] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4644.083837] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4644.151972] Lustre: lustre-MDT0000: new disk, initializing [ 4644.214617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4644.218889] Lustre: Skipped 5 previous similar messages [ 4644.232364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4648.797425] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 4660.178990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4660.338604] Lustre: 136141:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4660.387592] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4660.393556] Lustre: Skipped 1 previous similar message [ 4660.488323] Lustre: lustre-MDT0001: new disk, initializing [ 4660.671214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4660.687854] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4666.116697] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 4677.113648] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4685.728521] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4696.325847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4696.813313] Lustre: lustre-OST0000: new disk, initializing [ 4696.822942] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4696.835229] Lustre: 138108:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4698.888461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4698.907196] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4699.033476] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4704.201531] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 4715.223376] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4724.292228] Lustre: lustre-MDD0000: changelog on [ 4726.870137] Lustre: lustre-MDD0001: changelog on [ 6746.940962] Lustre: lustre-MDD0001: changelog off [ 6749.024614] Lustre: 136146:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6749.844310] Lustre: lustre-MDD0000: changelog off [ 6757.349584] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6757.352785] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6757.360410] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6757.360420] Lustre: Skipped 3 previous similar messages [ 6757.364728] LustreError: Skipped 1 previous similar message [ 6757.404048] Lustre: Skipped 1 previous similar message [ 6762.467788] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6762.483156] Lustre: Skipped 2 previous similar messages [ 6767.590046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6767.607282] Lustre: Skipped 1 previous similar message [ 6771.167282] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6771.524687] Lustre: server umount lustre-OST0000 complete [ 6777.825665] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6777.849966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6777.852037] Lustre: Skipped 1 previous similar message [ 6777.874893] Lustre: Skipped 1 previous similar message [ 6782.948246] LustreError: 139566:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6782.971791] LustreError: 139566:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6783.280926] Lustre: server umount lustre-MDT0000 complete [ 6788.012782] LustreError: 136934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787476252 with bad export cookie 5384954547287721700 [ 6788.013285] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6788.036054] LustreError: 136934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6788.766240] Lustre: server umount lustre-MDT0001 complete [ 6803.139225] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 6806.292288] Key type lgssc unregistered [ 6806.651347] LNet: 147279:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6806.661850] LNetError: 147279:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6806.681560] LNet: Removed LNI 192.168.202.124@tcp [ 6807.668156] Key type .llcrypt unregistered [ 6807.671742] Key type ._llcrypt unregistered [ 6827.588757] Key type ._llcrypt registered [ 6827.591928] Key type .llcrypt registered [ 6827.661086] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [ 6843.240823] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 6844.516101] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6844.544173] alg: No test for adler32 (adler32-zlib) [ 6845.581866] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 6845.883305] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 6847.575225] Key type lgssc registered [ 6848.903046] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6896.204136] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6896.267836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6897.684885] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6897.730577] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6897.834260] Lustre: lustre-MDT0000: new disk, initializing [ 6897.909210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6897.927416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6904.170368] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 6920.079454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6920.441212] Lustre: 150486:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6920.593490] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6920.596377] Lustre: Skipped 1 previous similar message [ 6920.873316] Lustre: lustre-MDT0001: new disk, initializing [ 6921.094297] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6921.161100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6921.201335] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6927.564736] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 6939.293479] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6946.762163] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6955.253392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6955.642712] Lustre: lustre-OST0000: new disk, initializing [ 6955.651594] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6955.662495] Lustre: 152452:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6955.789165] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6962.868176] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 6964.863529] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6964.875078] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6965.027974] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6972.887350] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6978.628288] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_module obdecho/obdecho [ 6980.663237] LustreError: 153458:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6980.673595] LustreError: 153458:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6982.469045] LustreError: 153554:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6997.986873] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6997.999252] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6998.018507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7001.057194] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7001.058327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7003.317770] Lustre: server umount lustre-MDT0000 complete [ 7007.537532] LustreError: 152451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787476471 with bad export cookie 16298194945476311538 [ 7007.546908] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7007.557547] LustreError: 152451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7007.803760] Lustre: server umount lustre-MDT0001 complete [ 7022.468037] Lustre: server umount lustre-OST0000 complete [ 7043.133417] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [ 7054.879300] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 7097.568454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7098.026327] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7098.074855] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7098.222469] Lustre: lustre-MDT0000: new disk, initializing [ 7098.417418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7098.469229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7103.333393] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7116.149361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7116.328572] Lustre: 157273:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7116.397825] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7116.403737] Lustre: Skipped 1 previous similar message [ 7116.633279] Lustre: lustre-MDT0001: new disk, initializing [ 7116.832428] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7116.909954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7116.926709] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7122.108176] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7132.313865] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7139.194841] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7148.116542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7148.590585] Lustre: lustre-OST0000: new disk, initializing [ 7148.600590] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7148.608876] Lustre: 159242:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7148.766815] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7154.755639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7154.769331] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7154.965954] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7156.083852] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7166.608687] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7172.579697] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7172.809738] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7179.422950] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7179.645682] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7185.890355] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7185.900546] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7185.918537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7185.921865] Lustre: Skipped 2 previous similar messages [ 7185.928782] Lustre: Skipped 2 previous similar messages [ 7187.919390] Lustre: server umount lustre-OST0000 complete [ 7193.569472] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7193.578987] LustreError: Skipped 1 previous similar message [ 7193.587817] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7193.598781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7193.601964] Lustre: Skipped 1 previous similar message [ 7198.367504] Lustre: server umount lustre-MDT0000 complete [ 7201.250394] LustreError: 158738:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7201.277694] LustreError: 158738:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7201.591414] LustreError: 157267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787476665 with bad export cookie 16298194945476312497 [ 7201.591966] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7201.603441] LustreError: 157267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7201.857127] Lustre: server umount lustre-MDT0001 complete [ 7205.303702] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 05:17:48 (1787476668) [ 7206.801541] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7208.364466] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 05:17:51 (1787476671) [ 7216.350274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7216.779389] LustreError: 161657:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7216.854272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7221.195597] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7222.244269] LustreError: 161658:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7227.364768] LustreError: 161657:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7229.417812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7234.781753] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7242.334322] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7248.504982] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7256.466828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7256.821075] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7256.829439] Lustre: Skipped 1 previous similar message [ 7262.683928] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7272.467482] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7284.398411] LustreError: 161684:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x240000bd0:0x2:0x0]: rc = -108 [ 7284.422515] Lustre: 161684:0:(llog_cat_server.c:670:llog_cat_cancel_arr_rec()) lustre-MDT0001-osp-MDT0000: fail to cancel 1 records in [0x240000bd0:0x2:0x0]: rc = -108 [ 7284.707716] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7284.713240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7284.725362] Lustre: Skipped 3 previous similar messages [ 7284.732066] Lustre: Skipped 3 previous similar messages [ 7290.970681] Lustre: server umount lustre-MDT0000 complete [ 7293.955072] LustreError: 161654:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7295.113190] LustreError: 161638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787476759 with bad export cookie 16298194945476313379 [ 7295.114920] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7295.125233] LustreError: 161638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7299.109425] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.24@tcp (stopping) [ 7299.112940] Lustre: Skipped 5 previous similar messages [ 7300.065582] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7301.713587] Lustre: server umount lustre-MDT0001 complete [ 7308.863730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7320.717729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7321.001614] LustreError: 165499:0:(llog.c:1655:llog_backup()) MGC192.168.202.124@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7321.007134] Lustre: 165499:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.124@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7329.877353] LustreError: 165503:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7329.904091] LustreError: 165503:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7346.179070] LustreError: 165505:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7346.216379] LustreError: 165505:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 7380.995812] LustreError: 165525:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7381.003393] LustreError: 165525:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 24 previous similar messages [ 7381.988925] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7381.996353] Lustre: 165499:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.124@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7422.967554] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7428.072850] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7428.229387] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7439.677828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7440.927327] LustreError: 166242:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 7440.942642] LustreError: 166242:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@00000000204ae435 x1874304759365248/t0(0) o250->MGC192.168.202.124@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1787476905 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7440.987206] LustreError: 166242:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 7447.027859] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7448.949024] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7458.581402] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7460.508758] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7467.558293] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7469.301851] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7492.327250] Lustre: server umount lustre-MDT0000 complete [ 7493.094226] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7493.102608] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7493.119306] LustreError: 165503:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7493.138775] LustreError: 165503:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 47 previous similar messages [ 7496.212265] LustreError: 161637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787476960 with bad export cookie 16298194945476319833 [ 7496.215099] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7496.219028] LustreError: 161637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7496.521673] Lustre: server umount lustre-MDT0001 complete [ 7510.827064] Lustre: server umount lustre-OST0000 complete [ 7534.168800] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [ 7545.974648] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 7587.884759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7588.251325] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7588.319360] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7588.454762] Lustre: lustre-MDT0000: new disk, initializing [ 7588.613918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7588.619974] Lustre: Skipped 1 previous similar message [ 7588.656734] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7596.190591] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7610.956863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7611.111246] Lustre: 170608:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7611.526284] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7616.606354] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7626.532027] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7633.231194] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7640.491989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7640.878072] Lustre: lustre-OST0000: new disk, initializing [ 7640.886789] Lustre: Skipped 1 previous similar message [ 7640.900435] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7640.912971] Lustre: Skipped 2 previous similar messages [ 7640.930141] Lustre: 172576:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7647.392653] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7648.362658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7648.374121] Lustre: Skipped 1 previous similar message [ 7648.388109] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7648.488124] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7657.866418] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7663.580638] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7663.974958] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7668.196411] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7668.495195] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7674.341479] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7674.345358] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7674.355224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7674.355238] Lustre: Skipped 1 previous similar message [ 7674.367051] LustreError: Skipped 1 previous similar message [ 7674.383880] Lustre: Skipped 3 previous similar messages [ 7677.365289] Lustre: server umount lustre-OST0000 complete [ 7683.041522] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7697.377324] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7697.738638] Lustre: server umount lustre-MDT0000 complete [ 7699.940815] LustreError: 170618:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7699.963388] LustreError: 170618:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7701.831706] LustreError: 170600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787477166 with bad export cookie 16298194945476336668 [ 7701.836196] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7701.846877] LustreError: 170600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7702.148764] Lustre: server umount lustre-MDT0001 complete [ 7712.524940] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 05:26:15 (1787477175) [ 7719.036442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7724.882866] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7735.565872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7741.573530] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7749.456670] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7756.407042] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7765.449921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7774.934082] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7784.319381] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7800.539257] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 05:27:44 (1787477264) [ 7804.899487] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7804.903436] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7804.916502] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7804.925266] Lustre: Skipped 3 previous similar messages [ 7804.936971] Lustre: Skipped 9 previous similar messages [ 7810.751849] Lustre: server umount lustre-OST0000 complete [ 7822.455115] Lustre: server umount lustre-MDT0000 complete [ 7827.280181] LustreError: 174970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787477291 with bad export cookie 16298194945476337550 [ 7827.287961] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7827.290716] LustreError: 174970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7827.822255] Lustre: server umount lustre-MDT0001 complete [ 7841.815613] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 7844.644520] Key type lgssc unregistered [ 7845.066260] LNet: 178726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7845.073539] LNetError: 178726:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7845.085537] LNet: Removed LNI 192.168.202.124@tcp [ 7845.783847] Key type .llcrypt unregistered [ 7845.786869] Key type ._llcrypt unregistered [ 7867.392350] Key type ._llcrypt registered [ 7867.396185] Key type .llcrypt registered [ 7867.615716] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 7869.383538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7869.716451] alg: No test for adler32 (adler32-zlib) [ 7870.892443] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 7871.172986] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 7872.991294] Key type lgssc registered [ 7874.445993] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7882.702410] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7882.728656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7898.208234] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7903.327398] LustreError: 179969:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.124@tcp: failed processing log, type 4: rc = -110 [ 7908.447402] LustreError: 179969:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7908.464177] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7908.468762] LustreError: MGC192.168.202.124@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7908.490374] LustreError: 179969:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7908.497306] LustreError: 179969:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7908.503520] LustreError: 179969:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7908.583648] Lustre: server umount lustre-OST0000 complete [ 7908.590889] LustreError: 179969:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7917.735688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7919.147916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7923.812212] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7929.320677] LustreError: 180468:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7929.348331] LustreError: 180468:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7934.139857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7934.445392] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7939.196503] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7947.654986] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7953.535409] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7960.176869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7960.356726] LustreError: 182286:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7960.372770] Lustre: 182286:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.202.124@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7960.543301] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7966.658859] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 7974.750479] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7979.811131] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7979.816677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7979.833367] Lustre: Skipped 2 previous similar messages [ 7979.844044] Lustre: Skipped 2 previous similar messages [ 7985.123758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7985.128774] Lustre: Skipped 2 previous similar messages [ 7990.243083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7990.275577] Lustre: Skipped 2 previous similar messages [ 7991.264947] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7991.548409] Lustre: server umount lustre-MDT0000 complete [ 7995.199110] LustreError: 180449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787477459 with bad export cookie 13083016853117623 [ 7995.207515] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7995.214872] LustreError: 180449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7995.365208] LustreError: 180463:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7995.367235] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7995.423502] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7996.255369] LustreError: 181199:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000c68169d7 x1874305834054784/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7996.389956] Lustre: server umount lustre-MDT0001 complete [ 8002.680508] Lustre: server umount lustre-OST0000 complete [ 8011.780328] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [ 8022.321484] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 8056.764123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8056.978736] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8057.004340] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8057.059380] Lustre: lustre-MDT0000: new disk, initializing [ 8057.122259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8057.136389] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8061.463324] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 8075.160515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8075.371709] Lustre: 185810:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8075.447803] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8075.457177] Lustre: Skipped 1 previous similar message [ 8075.621353] Lustre: lustre-MDT0001: new disk, initializing [ 8075.757152] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8075.801102] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8075.822244] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8081.447739] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 8091.598136] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8097.735298] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8105.059991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8105.301388] Lustre: lustre-OST0000: new disk, initializing [ 8105.304397] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8105.309566] Lustre: 187779:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8105.411236] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8112.559231] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 8115.317520] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8115.327554] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8115.461172] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8123.772303] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8130.391168] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8130.679610] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8135.582148] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8135.856558] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8141.310108] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8141.315992] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8141.319202] LustreError: Skipped 1 previous similar message [ 8141.324805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8141.353955] Lustre: Skipped 1 previous similar message [ 8143.815878] Lustre: server umount lustre-OST0000 complete [ 8151.532845] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8151.538306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8151.560067] Lustre: Skipped 1 previous similar message [ 8151.576863] Lustre: Skipped 2 previous similar messages [ 8154.122510] Lustre: server umount lustre-MDT0000 complete [ 8156.642307] LustreError: 185824:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8156.663411] LustreError: 185824:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8159.387603] LustreError: 185805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787477623 with bad export cookie 13083016853118337 [ 8159.388219] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8159.402259] LustreError: 185805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8159.832492] Lustre: server umount lustre-MDT0001 complete [ 8170.024849] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 05:33:52 (1787477632) [ 8192.251445] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 8195.815300] Key type lgssc unregistered [ 8196.288395] LNet: 190731:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8196.299533] LNetError: 190731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8196.332745] LNet: Removed LNI 192.168.202.124@tcp [ 8197.409616] Key type .llcrypt unregistered [ 8197.413969] Key type ._llcrypt unregistered [ 8218.262397] Key type ._llcrypt registered [ 8218.265644] Key type .llcrypt registered [ 8218.378194] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 8219.349459] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8219.498308] alg: No test for adler32 (adler32-zlib) [ 8220.474326] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 8220.679103] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 8222.401950] Key type lgssc registered [ 8223.614850] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8231.342789] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8231.360398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8232.923834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8237.111586] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 8243.175441] LustreError: 191950:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8243.196633] LustreError: 191950:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8246.153807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8246.427924] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8251.222643] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 8260.319673] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8267.054732] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8275.186451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8275.409184] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8275.420214] LustreError: 193816:0:(llog.c:1696:llog_backup()) MGC192.168.202.124@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8275.426402] Lustre: 193816:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.202.124@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8275.578758] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8281.920601] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 8290.045909] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8292.322616] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8292.337080] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8292.359148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8296.417349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8296.420951] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8296.428378] Lustre: Skipped 1 previous similar message [ 8296.441732] Lustre: Skipped 1 previous similar message [ 8298.469204] Lustre: server umount lustre-MDT0000 complete [ 8301.543663] LustreError: 191950:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8301.570325] LustreError: 191950:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8302.290780] LustreError: 191931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787477766 with bad export cookie 3340987551338389285 [ 8302.315502] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8302.317255] LustreError: 191931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8302.720253] Lustre: server umount lustre-MDT0001 complete [ 8316.212445] Lustre: server umount lustre-OST0000 complete [ 8327.067281] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [ 8338.506535] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 8378.860054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8379.149444] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8379.184534] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8379.237361] Lustre: lustre-MDT0000: new disk, initializing [ 8379.299766] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8379.337890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8383.677804] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 8393.891756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8393.991297] Lustre: 197341:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8394.024642] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8394.027784] Lustre: Skipped 1 previous similar message [ 8394.030391] LustreError: 198123:0:(llog.c:1696:llog_backup()) MGC192.168.202.124@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8394.036204] LustreError: 198123:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8394.040807] Lustre: 198123:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.124@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8394.054235] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8394.057852] Lustre: Skipped 1 previous similar message [ 8394.119846] Lustre: lustre-MDT0001: new disk, initializing [ 8394.210691] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8394.246263] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8394.265469] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8398.188358] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 8406.991363] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8412.758483] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8421.054108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8421.722218] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8421.729018] Lustre: Skipped 1 previous similar message [ 8421.737054] LustreError: 199284:0:(llog.c:1696:llog_backup()) MGC192.168.202.124@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8421.756866] LustreError: 199284:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8421.781524] Lustre: 199284:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.124@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8421.800854] Lustre: 199284:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8421.841262] Lustre: lustre-OST0000: new disk, initializing [ 8421.859653] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8421.885207] Lustre: 199308:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8422.040374] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8428.086094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8428.099429] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8428.241206] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8430.419449] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 8440.603512] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8445.252485] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8445.437865] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8449.968769] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8450.197731] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8454.113925] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8454.123341] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8454.124585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8454.147242] Lustre: Skipped 1 previous similar message [ 8454.160051] Lustre: Skipped 2 previous similar messages [ 8458.329935] Lustre: server umount lustre-OST0000 complete [ 8464.353953] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8464.377898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8469.474826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8469.480977] Lustre: Skipped 2 previous similar messages [ 8476.127272] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8476.391657] Lustre: server umount lustre-MDT0000 complete [ 8479.713113] LustreError: 197348:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8479.736694] LustreError: 197348:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8481.100590] LustreError: 197332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787477945 with bad export cookie 3340987551338390006 [ 8481.107034] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8481.112790] LustreError: 197332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8481.959203] Lustre: server umount lustre-MDT0001 complete [ 8493.850672] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 05:39:17 (1787477957) [ 8512.493931] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 8522.628664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8523.151247] LustreError: 202438:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8523.220923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8528.355281] LustreError: 202439:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8528.426832] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 8537.658412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8542.780638] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 8546.105194] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8552.342634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8558.853925] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 8567.958891] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8568.266903] Lustre: lustre-OST0001: new disk, initializing [ 8568.269411] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8568.285664] Lustre: 204745:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8568.472596] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8568.475449] Lustre: Skipped 2 previous similar messages [ 8574.773321] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 8575.633444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8575.639642] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8575.809946] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8586.317934] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8593.597866] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8595.830734] Lustre: DEBUG MARKER: ADD OST3 [ 8601.356816] loop: module loaded [ 8601.490317] loop0: detected capacity change from 0 to 204800000 [ 8601.837031] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8601.848595] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8601.859081] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8608.973699] loop0: detected capacity change from 0 to 204800000 [ 8610.331855] Lustre: DEBUG MARKER: STOP OST3 [ 8616.537829] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8616.738719] Lustre: lustre-OST0003: new disk, initializing [ 8616.744819] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8616.753630] Lustre: 206740:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8618.024135] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8618.215153] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8618.226812] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8618.309185] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8620.064156] LustreError: 206826:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8622.768788] Lustre: Failing over lustre-OST0003 [ 8623.056087] Lustre: server umount lustre-OST0003 complete [ 8623.167127] LustreError: 203924:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.202.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8623.190284] LustreError: 203924:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8623.601803] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8623.614051] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8623.626089] Lustre: Skipped 1 previous similar message [ 8623.658690] LustreError: Skipped 1 previous similar message [ 8626.859244] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 8636.784752] Lustre: DEBUG MARKER: START OST3 again [ 8643.589561] LustreError: 203925:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.202.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8643.620824] LustreError: 203925:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 8646.109545] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8646.554978] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8646.570755] Lustre: Skipped 1 previous similar message [ 8646.619551] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8647.732387] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8647.996185] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8647.998958] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8657.107354] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 8680.564287] Lustre: Failing over lustre-OST0003 [ 8680.822947] Lustre: server umount lustre-OST0003 complete [ 8684.002307] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8684.010099] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8684.016669] LustreError: Skipped 1 previous similar message [ 8684.020056] LustreError: 203923:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8684.038972] Lustre: Skipped 2 previous similar messages [ 8684.060563] LustreError: 203923:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8700.898741] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8700.908372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8700.920329] Lustre: Skipped 3 previous similar messages [ 8700.941142] Lustre: Skipped 5 previous similar messages [ 8706.629381] Lustre: server umount lustre-MDT0000 complete [ 8710.076863] LustreError: 202419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787478174 with bad export cookie 3340987551338390888 [ 8710.080077] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8710.086385] LustreError: 202419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8716.492351] Lustre: server umount lustre-MDT0001 complete [ 8720.996687] Lustre: server umount lustre-OST0000 complete [ 8724.716753] Lustre: server umount lustre-OST0001 complete [ 8730.231295] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [ 8739.362938] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 8786.175975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8786.688446] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8786.806561] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8787.008539] Lustre: lustre-MDT0000: new disk, initializing [ 8787.178355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8787.199585] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8792.394838] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 8804.134618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8804.359329] Lustre: 212213:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8804.428846] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8804.432363] Lustre: Skipped 1 previous similar message [ 8804.496431] Lustre: lustre-MDT0001: new disk, initializing [ 8804.696659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8804.715033] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8810.424980] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 8819.922669] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8825.935524] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8833.618044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8833.881356] Lustre: 214184:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8835.913769] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8836.086182] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8839.922941] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 8849.994313] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8854.964419] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8855.164118] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8860.053290] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8860.227697] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8866.796149] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8866.798654] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8866.828422] Lustre: Skipped 2 previous similar messages [ 8866.833674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8866.853791] Lustre: Skipped 9 previous similar messages [ 8868.694302] Lustre: server umount lustre-OST0000 complete [ 8876.522834] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8879.531489] Lustre: server umount lustre-MDT0000 complete [ 8882.145499] LustreError: 212219:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8882.167921] LustreError: 212219:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 8883.599410] LustreError: 212204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787478347 with bad export cookie 3340987551338393177 [ 8883.621233] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8883.621823] LustreError: 212204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8884.001320] Lustre: server umount lustre-MDT0001 complete [ 8888.627702] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 05:45:51 (1787478351) [ 8909.174927] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [ 8920.301466] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 8956.175394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8956.613197] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8956.675191] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8956.688967] Lustre: Skipped 1 previous similar message [ 8956.884877] Lustre: lustre-MDT0000: new disk, initializing [ 8956.889849] Lustre: Skipped 1 previous similar message [ 8957.068068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8957.076037] Lustre: Skipped 1 previous similar message [ 8962.661558] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 8975.321389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8975.501763] Lustre: 218796:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8975.967221] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8981.415191] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 8994.884845] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9000.691505] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9007.622131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9008.009967] Lustre: 220766:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9009.468470] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9009.563563] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9014.436333] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 9023.870744] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9028.339353] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9028.510657] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9033.161764] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9033.421644] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9038.315374] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9038.356904] Lustre: Skipped 3 previous similar messages [ 9038.360677] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9038.368984] Lustre: Skipped 4 previous similar messages [ 9041.620376] Lustre: server umount lustre-OST0000 complete [ 9047.528822] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9051.762345] Lustre: server umount lustre-MDT0000 complete [ 9053.667057] LustreError: 218803:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9053.681692] LustreError: 218803:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9055.961480] LustreError: 218788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787478520 with bad export cookie 3340987551338394066 [ 9055.965240] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9055.970741] LustreError: 218788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9056.390756] Lustre: server umount lustre-MDT0001 complete [ 9065.395877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9065.855189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9065.859285] Lustre: Skipped 5 previous similar messages [ 9070.375771] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 9078.444418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9082.762746] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 9090.283816] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9095.828529] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9101.883686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9108.408340] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 9116.303399] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9126.890930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9126.912498] Lustre: Skipped 3 previous similar messages [ 9132.736063] Lustre: server umount lustre-OST0000 complete [ 9142.511773] Lustre: server umount lustre-MDT0000 complete [ 9146.682469] LustreError: 222962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787478610 with bad export cookie 3340987551338394948 [ 9146.694647] LustreError: 222962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9146.715107] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9147.445291] Lustre: server umount lustre-MDT0001 complete [ 9255.716795] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [ 9258.418303] Key type lgssc unregistered [ 9258.783992] LNet: 226647:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9258.812766] LNetError: 226647:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9259.880558] LNet: Removed LNI 192.168.202.124@tcp [ 9260.885182] Key type .llcrypt unregistered [ 9260.892665] Key type ._llcrypt unregistered [ 9290.857319] Key type ._llcrypt registered [ 9290.858859] Key type .llcrypt registered [ 9290.996803] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [ 9308.322702] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 9310.235834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9310.409627] alg: No test for adler32 (adler32-zlib) [ 9311.902356] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [ 9312.111041] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [ 9313.783523] Key type lgssc registered [ 9315.925709] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9363.926649] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9363.955695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9365.299229] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9365.318693] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9365.365928] Lustre: lustre-MDT0000: new disk, initializing [ 9365.441096] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9365.464397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9370.621653] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 9382.243799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9382.387199] Lustre: 230005:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9382.439220] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9382.443109] Lustre: Skipped 1 previous similar message [ 9382.522282] Lustre: lustre-MDT0001: new disk, initializing [ 9382.652347] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9382.687025] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9382.701113] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9388.203177] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 9400.022846] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9407.528770] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9415.911598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9416.157843] Lustre: lustre-OST0000: new disk, initializing [ 9416.164362] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9416.171245] Lustre: 231972:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9416.249856] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9421.882956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9421.899707] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9422.010456] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9422.844672] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 9433.510212] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9438.917141] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9439.177456] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9443.495326] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9443.729224] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9447.907772] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9447.913407] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9447.922068] LustreError: Skipped 1 previous similar message [ 9447.923872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9447.949152] Lustre: Skipped 1 previous similar message [ 9451.795429] Lustre: server umount lustre-OST0000 complete [ 9458.151103] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9458.162385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9458.165239] Lustre: Skipped 2 previous similar messages [ 9461.739769] Lustre: server umount lustre-MDT0000 complete [ 9465.405249] LustreError: 229995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787478929 with bad export cookie 2081626326876910601 [ 9465.413670] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9465.418468] LustreError: 229995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9466.040877] Lustre: server umount lustre-MDT0001 complete [ 9487.449137] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 9497.461636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9497.886242] LustreError: 234907:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9497.925175] LustreError: 234907:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9497.987649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9502.664816] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 9503.205893] LustreError: 234908:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9508.339387] LustreError: 234907:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9510.898746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9511.096791] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9515.577723] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 9518.364270] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9524.717294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9525.011492] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9531.181411] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 9539.537331] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9539.718555] Lustre: lustre-OST0001: new disk, initializing [ 9539.722464] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9539.727549] Lustre: 237215:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9545.716155] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 9548.360567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9548.377193] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9548.445260] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9558.374416] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9565.378642] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9579.837919] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 05:57:22 (1787479042) [ 9583.072219] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9583.078349] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9583.088120] Lustre: Skipped 1 previous similar message [ 9583.094223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9589.753460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9589.757254] Lustre: Skipped 7 previous similar messages [ 9594.877815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9594.886896] Lustre: Skipped 3 previous similar messages [ 9596.383135] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9596.788769] Lustre: server umount lustre-MDT0000 complete [ 9599.982730] LustreError: 235635:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9600.010176] LustreError: 235635:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9601.139512] LustreError: 234888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787479065 with bad export cookie 2081626326876911483 [ 9601.140728] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9601.164301] LustreError: 234888:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9601.524746] Lustre: server umount lustre-MDT0001 complete [ 9616.453965] Lustre: server umount lustre-OST0000 complete [ 9630.260155] Lustre: server umount lustre-OST0001 complete [ 9637.700415] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [ 9649.380551] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [ 9686.793615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9687.007651] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9687.040816] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9687.086540] Lustre: lustre-MDT0000: new disk, initializing [ 9687.142856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9687.147017] Lustre: Skipped 1 previous similar message [ 9687.161582] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9692.515182] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 9704.940884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9705.126613] Lustre: 241163:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9705.200687] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9705.208326] Lustre: Skipped 1 previous similar message [ 9705.423980] Lustre: lustre-MDT0001: new disk, initializing [ 9705.655611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9705.675267] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9711.470859] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 9721.568992] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9727.748382] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9734.737755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9735.110802] Lustre: lustre-OST0000: new disk, initializing [ 9735.114334] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9735.125617] Lustre: 243131:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9735.232427] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9735.238178] Lustre: Skipped 1 previous similar message [ 9736.318599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9736.329630] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9736.416868] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9741.284257] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 9750.524363] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9755.337693] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9755.592788] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9761.414456] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9761.626826] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9767.392505] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9767.398280] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9767.420070] LustreError: Skipped 1 previous similar message [ 9767.424131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9767.442527] Lustre: Skipped 4 previous similar messages [ 9767.480530] Lustre: Skipped 1 previous similar message [ 9778.146558] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9778.269579] Lustre: server umount lustre-OST0000 complete [ 9782.241100] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9782.246362] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9787.873995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9787.878908] Lustre: Skipped 7 previous similar messages [ 9788.490600] Lustre: server umount lustre-MDT0000 complete [ 9791.955312] LustreError: 241155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787479256 with bad export cookie 2081626326876913618 [ 9791.971499] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9792.220120] Lustre: server umount lustre-MDT0001 complete [ 9800.733204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9801.081833] LustreError: 245360:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9801.154102] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9805.867698] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 9811.430175] LustreError: 245361:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9811.454252] LustreError: 245361:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9814.726748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9819.713302] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 9827.965684] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9834.545660] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9843.130606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9849.223063] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [ 9856.859906] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10005.476724] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10005.485729] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10005.500022] Lustre: Skipped 1 previous similar message [10005.522772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10018.273570] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10018.436712] Lustre: server umount lustre-OST0000 complete [10023.910770] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10023.921154] Lustre: Skipped 2 previous similar messages [10028.637416] Lustre: server umount lustre-MDT0000 complete [10029.024288] LustreError: 246089:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10029.050455] LustreError: 246089:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10032.785147] LustreError: 245341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787479496 with bad export cookie 2081626326876914500 [10032.792144] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10032.803937] LustreError: 245341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [10033.210765] Lustre: server umount lustre-MDT0001 complete [10128.345911] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [10131.389795] Key type lgssc unregistered [10131.608795] LNet: 248976:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10131.622321] LNetError: 248976:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10132.655104] LNet: Removed LNI 192.168.202.124@tcp [10133.486406] Key type .llcrypt unregistered [10133.490577] Key type ._llcrypt unregistered [10160.341733] Key type ._llcrypt registered [10160.344466] Key type .llcrypt registered [10160.438374] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [10184.560398] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [10185.897370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10186.061981] alg: No test for adler32 (adler32-zlib) [10187.496221] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [10187.907914] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [10189.575836] Key type lgssc registered [10191.006699] Lustre: Echo OBD driver; http://www.lustre.org/ [10234.161750] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [10243.612061] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10243.642279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10244.901944] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10244.927211] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10244.992486] Lustre: lustre-MDT0000: new disk, initializing [10245.068680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10245.084583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10249.513357] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [10259.337351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10259.526457] Lustre: 253049:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10259.597577] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10259.603047] Lustre: Skipped 1 previous similar message [10259.700159] Lustre: lustre-MDT0001: new disk, initializing [10259.848258] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10259.888613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10259.901986] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10264.364485] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [10269.243368] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10276.453352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10276.640512] Lustre: lustre-OST0000: new disk, initializing [10276.643142] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10276.647953] Lustre: 254677:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10276.724810] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10278.284721] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10278.296132] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10278.482591] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10283.712790] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [10294.961221] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10295.264732] Lustre: lustre-OST0001: new disk, initializing [10295.268524] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10295.276216] Lustre: 255550:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10295.394158] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10300.510831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10300.520404] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10300.655721] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10301.892844] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [10313.909796] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10326.105726] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10341.856461] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10341.862152] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10341.863992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10341.881864] Lustre: Skipped 3 previous similar messages [10344.413710] Lustre: server umount lustre-MDT0000 complete [10349.061059] LustreError: 253040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787479813 with bad export cookie 1033142443765145934 [10349.070437] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10349.091872] LustreError: 253040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10349.590037] Lustre: server umount lustre-MDT0001 complete [10363.994155] Lustre: server umount lustre-OST0000 complete [10378.057927] Lustre: server umount lustre-OST0001 complete [10392.854680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10402.477312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10411.214223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10422.506404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10444.025612] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [10453.519426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10453.533158] Lustre: lustre-MDT0000: reset Object Index mappings [10453.893698] LustreError: 260565:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10453.924228] LustreError: 260565:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10454.012337] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10459.118635] LustreError: 260566:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10459.387449] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [10464.227778] LustreError: 260565:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10468.121525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10468.134770] Lustre: lustre-MDT0001: reset Object Index mappings [10468.583327] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10473.670331] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [10477.202633] Lustre: 261707:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10484.510115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10484.986672] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10491.900501] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [10493.167567] LustreError: 262061:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10498.536682] LustreError: 262064:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10499.637018] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10501.549668] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10507.285182] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10507.974705] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [10516.462711] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10519.996479] Lustre: 263577:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10527.723811] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10527.729469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10527.738590] Lustre: Skipped 3 previous similar messages [10527.749557] Lustre: Skipped 6 previous similar messages [10531.777456] Lustre: server umount lustre-MDT0000 complete [10532.842713] LustreError: 261773:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10532.869132] LustreError: 261773:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10535.906703] LustreError: 260547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787480000 with bad export cookie 1033142443765152724 [10535.928307] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10536.187774] Lustre: server umount lustre-MDT0001 complete [10550.537354] Lustre: server umount lustre-OST0000 complete [10553.823398] Lustre: 250555:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787480002/real 1787480002] req@ffff93694ad11880 x1874308263707904/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787480018 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10553.856325] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10554.637772] Lustre: server umount lustre-OST0001 complete [10574.765291] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 06:13:57 (1787480037) [10593.917682] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [10605.757421] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [10647.704286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10647.963952] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10647.983518] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10648.062638] Lustre: lustre-MDT0000: new disk, initializing [10648.137953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10648.143685] Lustre: Skipped 1 previous similar message [10648.154598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10653.867942] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [10666.568362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10666.701914] Lustre: 267848:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10666.796488] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10666.808868] Lustre: Skipped 1 previous similar message [10666.898822] Lustre: lustre-MDT0001: new disk, initializing [10667.104549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10667.130337] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10672.007939] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [10683.157988] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10690.057509] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10697.887654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10698.213773] Lustre: lustre-OST0000: new disk, initializing [10698.216634] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10698.233671] Lustre: 269818:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10699.604598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10699.613860] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10699.809537] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10705.115744] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [10715.316362] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10721.118684] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10721.364502] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10725.969608] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10726.181854] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10730.473467] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10730.476926] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10730.488504] LustreError: Skipped 1 previous similar message [10730.491705] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10730.498353] Lustre: Skipped 1 previous similar message [10730.515367] Lustre: Skipped 1 previous similar message [10734.647516] Lustre: server umount lustre-OST0000 complete [10740.705258] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10740.710876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10740.714207] Lustre: Skipped 1 previous similar message [10740.725521] Lustre: Skipped 1 previous similar message [10745.458700] Lustre: server umount lustre-MDT0000 complete [10745.851801] LustreError: 267860:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10745.882477] LustreError: 267860:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10749.648123] LustreError: 267839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787480213 with bad export cookie 1033142443765154817 [10749.653060] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10749.657902] LustreError: 267839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [10750.062719] Lustre: server umount lustre-MDT0001 complete [10770.804202] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [10781.420510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10781.872253] LustreError: 272753:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10781.978183] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10781.988852] Lustre: Skipped 2 previous similar messages [10786.211745] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [10794.531641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10799.407345] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [10802.486872] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10810.081526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10820.073603] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [10830.255938] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10830.465287] Lustre: lustre-OST0001: new disk, initializing [10830.471822] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10830.477239] Lustre: 275064:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10832.135731] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10832.147592] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10832.272032] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10838.535417] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [10849.278534] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10853.804776] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10859.789740] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10859.871083] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10866.656222] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10866.663780] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10866.681786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10872.691888] Lustre: server umount lustre-MDT0000 complete [10873.347940] LustreError: 273097:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10873.393211] LustreError: 273097:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10876.600985] LustreError: 275827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787480340 with bad export cookie 1033142443765155699 [10876.604963] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10876.611675] LustreError: 275827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10876.897352] Lustre: server umount lustre-MDT0001 complete [10891.265679] Lustre: server umount lustre-OST0000 complete [10894.303252] Lustre: 250554:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787480342/real 1787480342] req@ffff936a46b72a00 x1874308263818368/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787480358 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10894.328607] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10894.345290] Lustre: Skipped 3 previous similar messages [10895.187159] Lustre: server umount lustre-OST0001 complete [10911.487963] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [10921.847529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10922.265438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10922.268571] Lustre: Skipped 3 previous similar messages [10927.030874] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [10935.357296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10940.394296] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [10943.196199] Lustre: 279259:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10950.312559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10956.650571] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [10967.118450] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10967.422369] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10972.686459] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10975.581369] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [10984.800263] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10989.102527] Lustre: 281139:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10994.960980] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10995.222697] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [11006.679697] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 06:21:09 (1787480469) [11022.831188] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11022.843534] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11022.856843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11022.858804] Lustre: Skipped 4 previous similar messages [11025.202579] Lustre: server umount lustre-MDT0000 complete [11028.683486] LustreError: 279638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787480492 with bad export cookie 1033142443765162300 [11028.699805] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11028.964175] LustreError: 278114:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11028.981841] LustreError: 278114:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [11043.296884] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11043.307190] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11043.317843] Lustre: Skipped 11 previous similar messages [11043.577927] Lustre: server umount lustre-MDT0001 complete [11053.958610] Lustre: server umount lustre-OST0000 complete [11057.911518] Lustre: server umount lustre-OST0001 complete [11065.144352] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [11081.471952] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [11140.267691] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [11152.318047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11152.516502] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11152.536821] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11152.590475] Lustre: lustre-MDT0000: new disk, initializing [11152.673427] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11157.566412] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11168.840798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11169.021747] Lustre: 285410:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11169.534435] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11174.906185] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11186.539522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11186.732456] Lustre: 287041:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11186.830748] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11186.833020] Lustre: Skipped 5 previous similar messages [11188.754625] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11193.447985] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11204.276167] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11204.551706] Lustre: 287912:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11206.219351] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11206.247948] Lustre: Skipped 1 previous similar message [11206.336832] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11212.494943] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11223.711823] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11227.752144] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11227.759709] Lustre: Skipped 1 previous similar message [11245.442439] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 06:25:08 (1787480708) [11250.650751] Lustre: server umount lustre-MDT0000 complete [11251.689853] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11251.701111] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11251.720253] Lustre: Skipped 5 previous similar messages [11255.100391] LustreError: 285401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787480719 with bad export cookie 1033142443765169916 [11255.107697] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11255.115254] LustreError: 285401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11255.588535] Lustre: server umount lustre-MDT0001 complete [11269.753466] Lustre: server umount lustre-OST0000 complete [11284.466588] Lustre: server umount lustre-OST0001 complete [11292.975600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11293.383477] LustreError: 290161:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11293.401037] LustreError: 290161:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [11297.616102] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11306.580757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11311.243803] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11319.206855] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11325.731944] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11333.051112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11339.447881] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11346.882686] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11366.409840] loop1: detected capacity change from 0 to 204800000 [11366.703210] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11366.714199] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11366.730286] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11372.056368] loop1: detected capacity change from 0 to 204800000 [11379.966046] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11380.156206] LustreError: 290149:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs831: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11380.208952] LustreError: 290149:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs831 [11380.238679] LustreError: 290149:0:(mgs_handler.c:517:mgs_target_reg()) fs831-MDT0001: Error checking target index: rc = -13 [11380.272758] LustreError: fs831-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11380.290661] LustreError: 293880:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11380.294194] LustreError: 293880:0:(tgt_mount.c:2129:server_put_super()) no obd fs831-MDT0001 [11380.315063] LustreError: 293880:0:(tgt_mount.c:116:server_deregister_mount()) fs831-MDT0001 not registered [11380.361756] Lustre: server umount fs831-MDT0001 complete [11380.364491] LustreError: 293880:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11394.529391] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11394.738523] Lustre: Setting parameter fs831-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs831-MDT0001 [11394.863979] Lustre: srv-fs831-MDT0001: No data found on store. Initialize space. [11394.871854] Lustre: Skipped 4 previous similar messages [11394.982502] Lustre: fs831-MDT0001: new disk, initializing [11394.990201] Lustre: Skipped 3 previous similar messages [11395.204908] Lustre: srv-fs831-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11395.225495] Lustre: 294348:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs831-MDT0001-mdtlov: can't create new llog: rc = -115 [11395.243675] LustreError: 294348:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs831-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11400.993159] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11408.009430] Lustre: server umount fs831-MDT0001 complete [11418.605978] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11420.310449] loop1: detected capacity change from 0 to 204800000 [11425.644127] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11425.992694] Lustre: 295916:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs831-MDT0001-mdtlov: can't create new llog: rc = -115 [11426.008469] LustreError: 295916:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs831-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11430.826817] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11435.279600] Lustre: server umount fs831-MDT0001 complete [11445.221656] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11445.229438] LustreError: Skipped 1 previous similar message [11445.235772] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11445.258643] Lustre: Skipped 3 previous similar messages [11445.275358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11450.935365] Lustre: server umount lustre-MDT0000 complete [11459.121425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11459.261438] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11463.328441] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11464.707681] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11470.983266] loop1: detected capacity change from 0 to 204800000 [11476.156089] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11476.797925] Lustre: 298364:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs831-MDT0001-mdtlov: can't create new llog: rc = -115 [11476.808674] LustreError: 298364:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs831-MDT0001-osd: get update log duration 1, retries 0, failed: rc = -115 [11481.108977] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11485.522586] Lustre: server umount fs831-MDT0001 complete [11505.637354] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11505.648095] LustreError: Skipped 1 previous similar message [11510.756201] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11510.766947] Lustre: Skipped 8 previous similar messages [11511.721994] Lustre: server umount lustre-OST0000 complete [11521.555848] Lustre: server umount lustre-MDT0000 complete [11526.248445] LustreError: 290142:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787480990 with bad export cookie 1033142443765172597 [11526.255490] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11526.263144] LustreError: 290142:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11526.844758] Lustre: server umount lustre-MDT0001 complete [11539.563319] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [11542.299508] Key type lgssc unregistered [11542.635766] LNet: 300710:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11542.645458] LNetError: 300710:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11542.685096] LNet: Removed LNI 192.168.202.124@tcp [11543.543344] Key type .llcrypt unregistered [11543.547160] Key type ._llcrypt unregistered [11564.669691] Key type ._llcrypt registered [11564.671422] Key type .llcrypt registered [11564.796280] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [11566.028798] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11566.119547] alg: No test for adler32 (adler32-zlib) [11567.191263] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [11567.399391] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [11569.088921] Key type lgssc registered [11570.112248] Lustre: Echo OBD driver; http://www.lustre.org/ [11577.809200] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11577.826831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11579.194301] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11583.550662] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11589.604674] LustreError: 301983:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11589.617995] LustreError: 301983:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11592.358282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11592.629383] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11596.927881] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11604.432814] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11610.355146] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11618.025612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11618.511946] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11624.712268] LustreError: 303809:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11625.288740] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11630.052804] LustreError: 303811:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11634.997265] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11635.182896] LustreError: 303810:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11641.878889] LustreError: 304135:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11641.921631] LustreError: 304135:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11652.100510] LustreError: 303810:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11652.121018] LustreError: 303810:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11657.435625] loop1: detected capacity change from 0 to 204800000 [11657.658686] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11657.673769] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11657.687258] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11663.873816] loop1: detected capacity change from 0 to 204800000 [11671.346624] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11671.565766] Lustre: fs28540-OST0000: new disk, initializing [11671.569990] Lustre: srv-fs28540-OST0000: No data found on store. Initialize space. [11671.576491] Lustre: 305716:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11671.715638] Lustre: fs28540-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11672.588801] LustreError: 304135:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11672.613754] LustreError: 304135:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [11679.973594] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11690.384790] Lustre: server umount fs28540-OST0000 complete [11702.783399] loop1: detected capacity change from 0 to 204800000 [11703.793332] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11708.338762] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11708.437446] LustreError: 303809:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11708.467027] LustreError: 303809:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 20 previous similar messages [11708.540449] Lustre: fs28540-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11713.941695] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11719.145193] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11723.353662] loop2: detected capacity change from 0 to 204800000 [11723.550914] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11723.576094] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11723.605029] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11729.136432] loop2: detected capacity change from 0 to 204800000 [11734.997763] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11735.123042] LustreError: fs28540-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11735.141326] LustreError: fs28540-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11735.151346] LustreError: 308760:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11735.158155] LustreError: 308760:0:(tgt_mount.c:2129:server_put_super()) no obd fs28540-OST0001 [11735.161564] LustreError: 308760:0:(tgt_mount.c:116:server_deregister_mount()) fs28540-OST0001 not registered [11735.207819] Lustre: server umount fs28540-OST0001 complete [11735.212415] LustreError: 308760:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11748.102327] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11748.288147] Lustre: fs28540-OST0001: new disk, initializing [11748.299658] Lustre: srv-fs28540-OST0001: No data found on store. Initialize space. [11748.307341] Lustre: 309311:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11748.373133] Lustre: fs28540-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11755.322313] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11764.519224] Lustre: server umount fs28540-OST0000 complete [11774.834783] Lustre: server umount fs28540-OST0001 complete [11774.995379] LustreError: 303809:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11775.004341] LustreError: 303809:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [11788.663280] loop1: detected capacity change from 0 to 204800000 [11794.493730] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11794.686105] Lustre: fs28540-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11801.448098] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11808.715775] loop2: detected capacity change from 0 to 204800000 [11814.756045] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11821.499725] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [11830.908613] Lustre: server umount fs28540-OST0000 complete [11840.087766] Lustre: server umount fs28540-OST0001 complete [11855.328605] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11855.335597] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11855.348841] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11857.379681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11857.380534] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11857.381247] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11857.430097] LustreError: Skipped 1 previous similar message [11859.627860] Lustre: server umount lustre-OST0000 complete [11864.035166] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11864.053366] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11864.071973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11864.078683] Lustre: Skipped 1 previous similar message [11867.623181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11867.641248] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11871.754818] Lustre: server umount lustre-MDT0000 complete [11876.000329] LustreError: 301962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787481340 with bad export cookie 3514163787309127629 [11876.023228] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11876.312839] Lustre: server umount lustre-MDT0001 complete [11889.055726] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [11892.285976] Key type lgssc unregistered [11892.617571] LNet: 315237:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11892.635383] LNetError: 315237:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11892.648588] LNet: Removed LNI 192.168.202.124@tcp [11893.536235] Key type .llcrypt unregistered [11893.539844] Key type ._llcrypt unregistered [11931.564956] Key type ._llcrypt registered [11931.569410] Key type .llcrypt registered [11931.654906] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [11948.733062] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [11950.353111] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11950.541669] alg: No test for adler32 (adler32-zlib) [11951.993865] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [11952.227130] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [11953.879841] Key type lgssc registered [11955.352307] Lustre: Echo OBD driver; http://www.lustre.org/ [11994.830524] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11994.900637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11996.274242] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11996.306061] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11996.406910] Lustre: lustre-MDT0000: new disk, initializing [11996.494247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11996.513057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12001.491189] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [12012.532721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12012.657966] Lustre: 319097:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12012.715680] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12012.720324] Lustre: Skipped 1 previous similar message [12012.788665] Lustre: lustre-MDT0001: new disk, initializing [12012.876841] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12012.917036] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12012.925240] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12017.286971] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [12026.955798] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12032.878522] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12040.806668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12041.173726] Lustre: lustre-OST0000: new disk, initializing [12041.178629] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12041.187589] Lustre: 321065:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12041.298753] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12047.970946] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [12048.439244] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12048.450574] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12048.577262] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12058.085859] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12060.357244] Lustre: Mounted lustre-client - version 2.17.57_63_gaeb0ce0 [12061.685239] Lustre: Unmounted lustre-client [12074.465568] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12074.474307] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12074.491329] Lustre: Skipped 1 previous similar message [12074.494562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12074.506390] Lustre: Skipped 1 previous similar message [12076.002984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12076.006550] Lustre: Skipped 1 previous similar message [12077.529617] Lustre: server umount lustre-OST0000 complete [12084.717291] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12084.731988] LustreError: Skipped 1 previous similar message [12084.738318] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12084.753646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12088.267343] Lustre: server umount lustre-MDT0000 complete [12092.308941] LustreError: 319091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787481556 with bad export cookie 1728028251284077500 [12092.320587] LustreError: 319091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12092.321734] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12092.710398] Lustre: server umount lustre-MDT0001 complete [12102.600493] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [12104.903917] Key type lgssc unregistered [12121.570075] LNet: 323229:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12121.593884] LNetError: 323229:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12191.270274] LNet: Removed LNI 192.168.202.124@tcp [12192.010501] Key type .llcrypt unregistered [12192.013605] Key type ._llcrypt unregistered [12211.171084] Key type ._llcrypt registered [12211.173857] Key type .llcrypt registered [12211.283404] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [12225.916643] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [12227.115580] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12227.252816] alg: No test for adler32 (adler32-zlib) [12228.354812] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [12228.673410] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [12230.367165] Key type lgssc registered [12231.391507] Lustre: Echo OBD driver; http://www.lustre.org/ [12264.022673] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 06:42:07 (1787481727) [12272.641588] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12272.709833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12274.082939] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12274.126854] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12274.214454] Lustre: lustre-MDT0000: new disk, initializing [12274.306198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12274.323231] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12280.058873] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [12292.084213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12292.404934] Lustre: 326453:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12292.568578] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12292.581287] Lustre: Skipped 1 previous similar message [12292.658409] Lustre: lustre-MDT0001: new disk, initializing [12292.755735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12292.814503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12292.828978] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12298.507613] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [12310.227734] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12316.777873] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12325.000281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12325.319309] Lustre: lustre-OST0000: new disk, initializing [12325.327508] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12325.334616] Lustre: 328424:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12325.425080] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12332.579578] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [12334.667739] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12334.688623] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12334.840317] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12345.281550] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12365.796665] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12365.797740] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12365.799837] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12365.804910] Lustre: Skipped 1 previous similar message [12370.915609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12370.918482] Lustre: Skipped 2 previous similar messages [12371.482429] Lustre: server umount lustre-OST0000 complete [12380.128627] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12380.134747] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12380.144234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12381.160291] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12382.836287] Lustre: server umount lustre-MDT0000 complete [12386.677328] LustreError: 328423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787481850 with bad export cookie 3038791354423899195 [12386.680096] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12386.689780] LustreError: 328423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12386.985812] Lustre: server umount lustre-MDT0001 complete [12400.735531] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [12404.040379] Key type lgssc unregistered [12404.290574] LNet: 330336:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12404.299032] LNetError: 330336:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12404.317598] LNet: Removed LNI 192.168.202.124@tcp [12405.151326] Key type .llcrypt unregistered [12405.154813] Key type ._llcrypt unregistered [12425.344763] Key type ._llcrypt registered [12425.346800] Key type .llcrypt registered [12425.426590] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [12426.604125] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12426.654492] alg: No test for adler32 (adler32-zlib) [12427.763189] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [12427.996234] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [12429.736146] Key type lgssc registered [12431.380102] Lustre: Echo OBD driver; http://www.lustre.org/ [12441.533743] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12441.566581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12443.266503] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12448.283257] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [12452.832442] LustreError: 331615:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12452.843362] LustreError: 331615:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12456.744604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12457.138039] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12461.785382] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [12470.310868] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12476.317337] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12482.462723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12482.814965] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12489.944251] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [12498.727314] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12513.066193] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12515.159635] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12522.017050] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.24@tcp (stopping) [12522.994508] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12523.039677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12523.043116] Lustre: Skipped 2 previous similar messages [12523.083916] Lustre: Skipped 2 previous similar messages [12525.864299] Lustre: server umount lustre-MDT0000 complete [12527.114473] LustreError: 331610:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12528.119682] LustreError: 331612:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12528.143480] LustreError: 331612:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12530.261537] LustreError: 331596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787481994 with bad export cookie 179431506396453 [12530.274141] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12530.281078] LustreError: 331596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12532.233809] LustreError: 331610:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12532.258323] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.24@tcp (stopping) [12533.218612] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12535.265410] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12535.270446] Lustre: Skipped 1 previous similar message [12536.928259] Lustre: server umount lustre-MDT0001 complete [12541.686955] Lustre: server umount lustre-OST0000 complete [12551.592600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12556.280532] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [12558.867778] Lustre: 335798:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.124@tcp [12561.032163] Lustre: server umount MGS complete [12570.037539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12570.527786] LustreError: 336370:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12570.553407] LustreError: 336370:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12570.605873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12575.445689] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [12579.817828] LustreError: 336371:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12579.839876] LustreError: 336371:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12585.289747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12585.963176] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12591.042441] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [12600.666215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12601.038161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12609.251219] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [12618.331138] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12620.045619] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12627.203351] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12629.355706] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12635.121209] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [12635.130368] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12635.163629] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12636.765311] Lustre: server umount lustre-OST0000 complete [12641.264432] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12641.280360] Lustre: Skipped 2 previous similar messages [12646.371776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12646.380045] Lustre: Skipped 5 previous similar messages [12647.011185] Lustre: server umount lustre-MDT0000 complete [12651.493164] LustreError: 337098:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12651.539257] LustreError: 337098:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12651.673664] LustreError: 336351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787482115 with bad export cookie 179431506397293 [12651.687357] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12651.691319] LustreError: 336351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12652.231605] Lustre: server umount lustre-MDT0001 complete [12665.363643] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [12668.679533] Key type lgssc unregistered [12669.078767] LNet: 339811:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12669.083043] LNetError: 339811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12670.124109] LNet: Removed LNI 192.168.202.124@tcp [12671.000935] Key type .llcrypt unregistered [12671.004418] Key type ._llcrypt unregistered [12690.824528] Key type ._llcrypt registered [12690.827173] Key type .llcrypt registered [12691.016958] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [12708.067967] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [12709.059512] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12709.438927] alg: No test for adler32 (adler32-zlib) [12710.719797] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [12711.124087] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [12712.776036] Key type lgssc registered [12714.749239] Lustre: Echo OBD driver; http://www.lustre.org/ [12755.147300] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12755.211404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12756.539757] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12756.592540] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12756.692671] Lustre: lustre-MDT0000: new disk, initializing [12756.808717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12756.828741] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12762.216327] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [12773.746485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12773.922879] Lustre: 342979:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12773.989995] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12773.997643] Lustre: Skipped 1 previous similar message [12774.083939] Lustre: lustre-MDT0001: new disk, initializing [12774.251703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12774.317572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12774.339986] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12779.629747] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [12790.554355] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12797.498912] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12804.570399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12804.894116] Lustre: lustre-OST0000: new disk, initializing [12804.899371] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12804.906270] Lustre: 344946:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12805.000583] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12811.305996] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [12812.366939] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12812.380762] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12812.533190] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12821.659600] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12826.343382] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12826.563957] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12832.585816] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12832.898384] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12838.379503] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12838.380079] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12838.381811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12838.408750] Lustre: Skipped 1 previous similar message [12843.500166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12843.514895] Lustre: Skipped 2 previous similar messages [12848.613886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12848.624907] Lustre: Skipped 1 previous similar message [12849.119218] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12849.483076] Lustre: server umount lustre-OST0000 complete [12856.288446] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12856.298809] LustreError: Skipped 1 previous similar message [12856.304776] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12856.323031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12858.858805] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12860.152351] Lustre: server umount lustre-MDT0000 complete [12864.260802] LustreError: 342971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787482328 with bad export cookie 14271549572707923454 [12864.267698] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12864.271961] LustreError: 342971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12864.840471] Lustre: server umount lustre-MDT0001 complete [12873.732282] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 06:52:16 (1787482336) [12890.841326] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [12901.363754] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [12942.257847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12942.426647] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12942.449369] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12942.511598] Lustre: lustre-MDT0000: new disk, initializing [12942.607838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12942.620548] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12946.921584] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [12957.933585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12958.111463] Lustre: 349707:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12958.151310] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12958.158715] Lustre: Skipped 1 previous similar message [12958.263852] Lustre: lustre-MDT0001: new disk, initializing [12958.419486] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12958.484267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12958.496943] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12963.492210] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [12974.252197] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12980.648357] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12988.609947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12988.919464] Lustre: lustre-OST0000: new disk, initializing [12988.925296] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12988.931219] Lustre: 351672:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12989.017828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12996.710889] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [12997.220403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12997.240499] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12997.332060] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13008.068193] Lustre: DEBUG MARKER: oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13038.569323] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13038.573632] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13038.579449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13038.579462] Lustre: Skipped 2 previous similar messages [13038.583770] Lustre: Skipped 1 previous similar message [13048.810259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13048.812734] Lustre: Skipped 4 previous similar messages [13052.384649] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13052.573377] Lustre: server umount lustre-OST0000 complete [13059.046625] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13062.802270] Lustre: server umount lustre-MDT0000 complete [13064.165639] LustreError: 349713:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13064.177281] LustreError: 349713:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13066.606385] LustreError: 349700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787482530 with bad export cookie 14271549572707924336 [13066.618089] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13066.621312] LustreError: 349700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13066.883074] Lustre: server umount lustre-MDT0001 complete [13077.621639] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [13080.454720] Key type lgssc unregistered [13080.714198] LNet: 353834:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13080.726847] LNetError: 353834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13081.766753] LNet: Removed LNI 192.168.202.124@tcp [13082.756663] Key type .llcrypt unregistered [13082.759821] Key type ._llcrypt unregistered [13169.132678] Key type ._llcrypt registered [13169.134891] Key type .llcrypt registered [13169.235101] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [13169.564503] Key type .llcrypt unregistered [13169.567973] Key type ._llcrypt unregistered [13235.706518] Key type ._llcrypt registered [13235.711820] Key type .llcrypt registered [13235.858313] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [13236.148513] Key type .llcrypt unregistered [13236.153848] Key type ._llcrypt unregistered [13260.319372] Key type ._llcrypt registered [13260.321232] Key type .llcrypt registered [13260.394338] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [13260.704510] Key type .llcrypt unregistered [13260.708648] Key type ._llcrypt unregistered [13285.008336] Key type ._llcrypt registered [13285.012596] Key type .llcrypt registered [13285.156515] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [13285.382594] Key type .llcrypt unregistered [13285.388466] Key type ._llcrypt unregistered [13349.234134] Key type ._llcrypt registered [13349.237027] Key type .llcrypt registered [13349.351951] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [13349.696530] Key type .llcrypt unregistered [13349.698295] Key type ._llcrypt unregistered [13373.627974] Key type ._llcrypt registered [13373.632525] Key type .llcrypt registered [13373.725446] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [13373.985450] Key type .llcrypt unregistered [13373.988223] Key type ._llcrypt unregistered [13400.152567] Key type ._llcrypt registered [13400.158365] Key type .llcrypt registered [13400.269327] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [13400.483342] Key type .llcrypt unregistered [13400.486919] Key type ._llcrypt unregistered [13425.182924] Key type ._llcrypt registered [13425.189849] Key type .llcrypt registered [13425.272860] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [13425.536423] Key type .llcrypt unregistered [13425.540251] Key type ._llcrypt unregistered [13449.812616] Key type ._llcrypt registered [13449.815738] Key type .llcrypt registered [13449.908797] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [13450.143297] Key type .llcrypt unregistered [13450.146026] Key type ._llcrypt unregistered [13473.407366] Key type ._llcrypt registered [13473.410197] Key type .llcrypt registered [13473.495779] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [13473.681450] Key type .llcrypt unregistered [13473.683100] Key type ._llcrypt unregistered [13518.365681] Key type ._llcrypt registered [13518.367217] Key type .llcrypt registered [13518.458715] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [13518.721146] Key type .llcrypt unregistered [13518.726277] Key type ._llcrypt unregistered [13539.828663] Key type ._llcrypt registered [13539.830356] Key type .llcrypt registered [13539.911406] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [13540.165887] Key type .llcrypt unregistered [13540.168596] Key type ._llcrypt unregistered [13559.915394] Key type ._llcrypt registered [13559.916675] Key type .llcrypt registered [13559.983320] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [13560.230540] Key type .llcrypt unregistered [13560.232604] Key type ._llcrypt unregistered [13594.668612] Key type ._llcrypt registered [13594.670381] Key type .llcrypt registered [13594.740092] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [13605.533883] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [13606.360498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13606.372151] alg: No test for adler32 (adler32-zlib) [13607.348513] Lustre: Lustre: Build Version: 2.17.57_63_gaeb0ce0 [13607.547564] LNet: Added LNI 192.168.202.124@tcp [8/256/0/180] [13609.183398] Key type lgssc registered [13610.146766] Lustre: Echo OBD driver; http://www.lustre.org/ [13635.971303] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13635.982846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13637.178476] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13637.206769] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13637.287837] Lustre: tst250fs-MDT0000: new disk, initializing [13637.345801] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13637.379769] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13640.925778] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [13649.055510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13649.133196] Lustre: 373904:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13649.175128] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13649.178537] Lustre: Skipped 1 previous similar message [13649.230815] Lustre: tst250fs-MDT0001: new disk, initializing [13649.302188] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13649.356305] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13649.382758] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13652.535105] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [13656.368131] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13660.921363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13661.116721] Lustre: tst250fs-OST0000: new disk, initializing [13661.120502] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13661.124879] Lustre: 375537:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13661.183941] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13665.769100] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [13667.891659] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13667.905499] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13667.982817] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13675.103282] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13675.242946] Lustre: tst250fs-OST0001: new disk, initializing [13675.246864] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13675.253690] Lustre: 376408:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13675.371286] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13679.750281] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_default_debug -1 all [13683.977413] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13684.778315] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13684.789445] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13684.883645] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13700.577129] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13700.586984] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13700.599560] Lustre: Skipped 3 previous similar messages [13700.620280] Lustre: Skipped 3 previous similar messages [13705.697440] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13705.700370] Lustre: Skipped 2 previous similar messages [13706.635238] Lustre: server umount tst250fs-MDT0000 complete [13709.220310] LustreError: 373898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787483173 with bad export cookie 9826853088569919496 [13709.226800] LustreError: MGC192.168.202.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13709.414737] Lustre: server umount tst250fs-MDT0001 complete [13722.162641] Lustre: server umount tst250fs-OST0000 complete [13734.748821] Lustre: server umount tst250fs-OST0001 complete [13745.895264] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [13751.118596] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [13770.970619] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 07:07:14 (1787483234) [13772.031263] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13783.749085] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing set_hostid [13790.674874] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing load_modules_local [13813.850873] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13523 sec ========= 07:07:57 (1787483277) [13814.859947] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 07:07:58 (1787483278) === [13817.431330] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 07:08:01 (1787483281) === [13846.810724] Lustre: DEBUG MARKER: oleg224-server.virtnet: executing unload_modules_local [13848.871509] Key type lgssc unregistered [13849.089117] LNet: 385550:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13849.095576] LNetError: 385550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13849.112403] LNet: Removed LNI 192.168.202.124@tcp [13849.721326] Key type .llcrypt unregistered [13849.725106] Key type ._llcrypt unregistered